Document classification device, document classification method, and document classification program

Abstract

In order to accurately and reliably classify a document to be classified, a document classification device (1) is provided with: a hypothetical sentence generation unit (11) that generates a plurality of hypothetical sentences, which are sentences related to a candidate classification into which a document is to be classified; an implication evaluation unit (12) that performs a process for evaluating, for each of the plurality of hypothetical sentences, whether the document implies the hypothetical sentence; and an evaluation result aggregation unit (13) that determines the classification into which the document is to be classified, on the basis of the result of the evaluation of each of the plurality of hypothetical sentences by the implication evaluation unit (12).

Description

[Technical Field]

[0001] The present invention relates to a document classification device that automatically classifies documents. [Background technology]

[0002] Recently, a large amount of data of various contents has been collected and accumulated, and accordingly, a technology for automatically classifying such data is required. For example, Non-Patent Document 1 below discloses a technology for automatically associating labels with text using a method called zero-shot classification.

[0003] More specifically, the technology of Non-Patent Document 1 first generates a premise sentence from the text to be classified, and also generates a hypothesis sentence related to the label of the candidate classification destination.The generated premise sentence and hypothesis sentence are then input into an implication model to determine the degree to which the label applies to the text to be classified.The implication model is a model constructed by machine learning to determine whether the premise sentence entails the hypothesis sentence, that is, whether they contain the same content. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Wenpeng Yin, Jamaal Hay, Dan Roth, "Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach", arXiv:1909.00161v1 [cs.CL], August 31, 2019 Summary of the Invention [Problem to be solved by the invention]

[0005] The technology of Non-Patent Document 1 has room for improvement in classification accuracy and stability, as the accuracy of the judgment depends on the type of hypothesis sentence corresponding to each label. For example, for the label "sports," the output value of the implication model will be different if a hypothesis sentence such as "this is a sentence about sports" is generated and if a hypothesis sentence such as "this refers to the topic of sports" is generated. Therefore, even for the same label "sports," the judgment result of the degree of fit will differ depending on which hypothesis sentence is generated.

[0006] One aspect of the present invention has been made in consideration of such problems, and one of its objectives is to provide a technology that enables documents to be classified stably and with high accuracy. [Means for solving the problem]

[0007] A document classification device according to one aspect of the present invention comprises: hypothesis sentence generation means for generating, for each candidate destination for a document, a plurality of hypothesis sentences that are sentences related to the candidate destination for the document; implication evaluation means for performing a process for each of the plurality of hypothesis sentences to evaluate whether the document entails the hypothesis sentence; and evaluation result collection means for determining a destination for the document based on the evaluation result by the implication evaluation means for each of the plurality of hypothesis sentences.

[0008] A document classification method according to one aspect of the present invention includes at least one processor generating, for each candidate destination for a document, a plurality of hypothesis sentences that are sentences related to the candidate destination; evaluating, for each of the hypothesis sentences, whether the document implies the hypothesis sentence; and determining a destination for the document based on the results of the evaluation for each of the hypothesis sentences.

[0009] A document classification program according to one aspect of the present invention causes a computer to function as: hypothesis sentence generation means for generating, for each candidate classification destination for a document, a plurality of hypothesis sentences that are sentences related to the candidate classification destination; implication evaluation means for performing a process for each of the plurality of hypothesis sentences to evaluate whether the document implies the hypothesis sentence; and evaluation result aggregation means for determining the classification destination for the document based on the results of evaluation by the implication evaluation means for each of the plurality of hypothesis sentences. [Effects of the Invention]

[0010] According to one aspect of the present invention, it is possible to stably classify documents to be classified with high accuracy. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a configuration of a document classification device according to a first exemplary embodiment of the present invention. [Figure 2] 1 is a flowchart showing the flow of a document classification method according to a first exemplary embodiment of the present invention. [Figure 3] FIG. 10 is a diagram showing an example of documents classified by the document classification method according to the second exemplary embodiment of the present invention. [Figure 4] FIG. 10 is a block diagram showing the configuration of a document classification device according to a second exemplary embodiment of the present invention. [Figure 5] FIG. 1 illustrates an example of a generation strategy. [Figure 6] FIG. 1 is a diagram illustrating a method for generating a language understanding model. [Figure 7] 10A and 10B are diagrams illustrating an example of a method for calculating the reliability of a generation strategy and a method for calculating a total score using the reliability. [Figure 8] FIG. 3 is a flowchart showing a flow of processing executed by the document classification device. [Figure 9] FIG. 1 is a diagram illustrating an example of a computer that executes instructions of a program, which is software that realizes the functions of each device according to each exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0012] Exemplary Embodiment 1 A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described below.

[0013] (Configuration of a document classification device) The configuration of a document classification device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the document classification device 1. As shown in Fig. 1, the document classification device 1 includes a hypothesis sentence generation unit 11, an implication evaluation unit 12, and an evaluation result aggregation unit 13.

[0014] The hypothesis sentence generation unit 11 generates, for each of the candidate classification destinations of the document, a plurality of hypothesis sentences that are sentences related to the candidate classification destination.

[0015] The implication evaluation unit 12 performs a process for evaluating whether the document entails the hypothesis sentence for each of the plurality of hypothesis sentences.

[0016] The evaluation result aggregating unit 13 determines the classification of the document based on the results of the evaluation by the implication evaluation unit 12 for each of the plurality of hypothesis sentences.

[0017] As described above, the document classification device 1 according to this exemplary embodiment is configured to include a hypothesis sentence generation unit 11 that generates, for each candidate destination for a document, a plurality of hypothesis sentences that are sentences related to the candidate destination, an implication evaluation unit 12 that performs a process for each of the hypothesis sentences to evaluate whether the document entails the hypothesis sentence, and an evaluation result aggregation unit 13 that determines the destination for the document based on the evaluation result of each of the hypothesis sentences by the implication evaluation unit 12. This configuration enables documents to be stably classified with high accuracy.

[0018] (Document classification program) The functions of the document classification device 1 described above can also be realized by a program. The document classification program according to this exemplary embodiment causes a computer to function as: a hypothesis sentence generation means that generates, for each candidate destination for a document, a plurality of hypothesis sentences that are sentences related to the candidate destination; an implication evaluation means that performs a process for evaluating, for each of the plurality of hypothesis sentences, whether the document entails the hypothesis sentence; and an evaluation result aggregation means that determines the destination for the document based on the evaluation result by the implication evaluation means for each of the plurality of hypothesis sentences. This document classification program enables documents to be stably classified with high accuracy.

[0019] (Document classification method flow) The flow of the document classification method according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the document classification method. Note that the execution entity of each step in this document classification method may be a processor provided in the document classification device 1, or a processor provided in another device, or each step may be executed by a processor provided in a different device.

[0020] In S11, at least one processor generates, for each of the classification destination candidates for the document, a plurality of hypothesis sentences that are sentences related to the classification destination candidate.

[0021] In S12, at least one processor performs a process for each of the plurality of hypothesis sentences to evaluate whether the document entails the hypothesis sentence.

[0022] In S13, at least one processor determines a classification for the document based on the results of the evaluation of each of the plurality of hypothesis sentences.

[0023] As described above, the document classification method according to this exemplary embodiment includes the steps of: generating, for each candidate destination for a document, a plurality of hypothesis sentences that are sentences related to the candidate destination for the document; evaluating, for each of the hypothesis sentences, whether the document entails the hypothesis sentence; and determining a destination for the document based on the evaluation results for each of the hypothesis sentences. This document classification method enables stable classification of documents to be classified with high accuracy.

[0024] Exemplary Embodiment 2 A second exemplary embodiment of the present invention will now be described in detail with reference to the drawings.

[0025] (Overview of document classification methods) An overview of the document classification method according to this exemplary embodiment (hereinafter referred to as the present method) will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of document classification using the present method. As shown in the figure, in the example of Fig. 3, a document x and a label set L to be classified are given as input data.

[0026] The classification destination can also be called a topic, and the classification of document x can be said to be the process of inferring the topic of document x. Also, if document x is extracted from a conversation and label set L is a set of labels indicating the speaker's emotions, the classification of document x can be said to be the estimation of the speaker's emotions. Also, if label set L is a set of labels indicating a situation, the classification of document x can be said to be the estimation of the situation indicated by document x.

[0027] Document x is a document to be classified, specifically, text data that reads, "I like beer. I have two Chihuahuas." Label set L indicates candidate classifications for document x, and label set L shown in Figure 3 includes three candidates: alcohol, sports, and pets. In Figure 3, the appropriateness of "alcohol" as a classification for document x is evaluated.

[0028] In this method, for each candidate classification of document x, multiple hypothesis sentences related to that candidate are generated. In the example of Figure 3, the text data "This is a sentence about alcohol" and the text data "This refers to the topic of alcohol" are generated as hypothesis sentences related to "alcohol."

[0029] Next, this method performs a process of evaluating the entailment between the hypothesis sentence and the document for each of the multiple hypothesis sentences. In the example of Figure 3, document x to be classified, "I like beer. I have two Chihuahuas," is evaluated to see if it entails the hypothesis sentence, "This is a sentence about alcohol," and an evaluation result of 0.93 is obtained. Similarly, document x is evaluated to see if it entails the hypothesis sentence, "This mentions the topic of alcohol," and an evaluation result of 0.47 is obtained.

[0030] Details will be explained later in the "Language Understanding Model" section, but these numbers indicate the degree to which document x implies the hypothesis sentence, with the closer the value is to 1 the higher the degree. Below, this number will be called the entailment score. Note that the degree to which document x implies the hypothesis sentence can be rephrased as the likelihood that the document implies the hypothesis sentence. The degree to which document x implies the hypothesis sentence can also be rephrased as the likelihood that the hypothesis sentence is correct when document x is used as the premise sentence.

[0031] If the hypothesis sentence and the target document x have the same meaning, or if the hypothesis sentence can be said to be correct when document x is used as the premise sentence, then it is highly likely that the target document candidate associated with the hypothesis sentence is suitable for the target document. For this reason, the implication score can be said to indicate the appropriateness of classifying the target document into the target document candidate.

[0032] For example, the entailment score between the hypothesis sentence "This is a sentence about alcohol" and the document x to be classified is 0.93. The entailment score of 0.93 is close to its maximum value of 1, and therefore indicates that document x is highly likely to entail the above hypothesis sentence. This entailment score also indicates that there is a high degree of validity in classifying document x as the classification candidate "alcohol," which is the basis for the hypothesis sentence "This is a sentence about alcohol."

[0033] On the other hand, a lower implication score of 0.47 was calculated for the hypothesis sentence "This is a sentence about alcohol," which was generated from the same classification candidate "alcohol." In this way, even if the combination of the document x to be classified and the classification candidate remains the same, the implication score calculated may differ depending on the generated hypothesis sentence.

[0034] Therefore, in this method, the evaluation results for each of the multiple hypothesis sentences generated as described above are aggregated to evaluate the appropriateness of classifying document x as a candidate for classification. In the example of Figure 3, the arithmetic mean of the calculated implication scores of 0.93 and 0.47 is calculated as a numerical value (hereinafter referred to as the overall score) indicating the appropriateness of classifying document x as "alcohol." This makes it possible to stably obtain highly accurate evaluation results of appropriateness compared to when generating only one hypothesis sentence.

[0035] By performing the above process for each candidate classification in the label set L, document x can be appropriately classified. For example, a candidate whose total score exceeds a preset threshold may be automatically determined as the classification destination, or the total score of each candidate may be output on a display device or the like, and the user may select the candidate to use as the classification destination for document x. Note that multiple classification destinations may be determined for one document. For example, for document x in FIG. 3, two classification destinations, "alcohol" and "pets," may be determined.

[0036] The determined classification destination can be recorded in association with document x. Document x associated with information indicating the classification destination can be used for searches, etc., expanding the scope of its use. Furthermore, document x associated with information indicating the classification destination can also be used as training data for machine learning to determine the classification destination of documents.

[0037] (Configuration of a document classification device) The configuration of a document classification device 2 according to this exemplary embodiment will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the document classification device 2. The document classification device 2 is a device for classifying documents, and as shown in the figure, includes a control unit 20 that controls each unit of the document classification device 2 in an integrated manner, and a storage unit 21 that stores various data used by the document classification device 2. The document classification device 2 also includes an input unit 22 that accepts user input operations to the document classification device 2, and an output unit 23 that outputs data from the document classification device 2. The document classification device 2 may be a device dedicated to document classification, or may be a general-purpose device that can be used for purposes other than document classification.

[0038] The control unit 20 also includes a data acquisition unit 201, a hypothesis sentence generation unit (hypothesis sentence generation means) 202, an implication evaluation unit (implication evaluation means) 203, an evaluation result aggregation unit (evaluation result aggregation means) 204, and a reliability calculation unit 205. The storage unit 21 also includes a generation strategy holding unit 211 and stores a language understanding model 212. The reliability calculation unit 205 will be described later in the section "Evaluation Taking Reliability into Account."

[0039] The data acquisition unit 201 acquires a document to be classified. The data acquisition unit 201 also acquires candidate classification destinations for classifying the document. For example, the data acquisition unit 201 may acquire text data input via the input unit 22 as the document to be classified, and may acquire a set of labels also input via the input unit 22 as candidate classification destinations.

[0040] For each of the classification destination candidates for the document, the hypothesis sentence generation unit 202 generates a plurality of hypothesis sentences which are sentences related to the classification destination candidate. More specifically, the hypothesis sentence generation unit 202 generates hypothesis sentences from the classification destination candidates acquired by the data acquisition unit 201, using a generation strategy recorded in the generation strategy holding unit 211 of the storage unit 21. A method for generating hypothesis sentences using a generation strategy will be described later in "Generation Strategy".

[0041] The implication evaluation unit 203 performs a process for each of a plurality of hypothesis sentences to evaluate whether the document to be classified entails the hypothesis sentence. More specifically, the implication evaluation unit 203 inputs a pair of the hypothesis sentence and document to be evaluated into the language understanding model 212 stored in the storage unit 21, and calculates an implication score, which is an index value indicating the degree to which the input document entails the input hypothesis sentence. Details of the language understanding model 212 will be explained later in "Language Understanding Model."

[0042] The evaluation result aggregating unit 204 determines the classification destination of the document to be classified based on the evaluation results by the implication evaluation unit 203 for each of the multiple hypothesis sentences. More specifically, the evaluation result aggregating unit 204 aggregates the implication scores calculated for each of the multiple hypothesis sentences to calculate an overall score indicating the evaluation result of the appropriateness of classifying the document to be classified into the candidate classification destination, and determines the classification destination using this overall score. Note that the overall score can be said to indicate the classification destination into which the document to be classified should be classified. Therefore, the evaluation result aggregating unit 204 may output the overall score as information indicating the classification destination of the document to be classified.

[0043] The method for calculating the overall score is not particularly limited as long as it is a method that calculates an overall score that reflects at least a portion of the calculated implication scores. For example, when the implication evaluation unit 203 uses the language understanding model 212 to calculate an implication score, which is an index value indicating the degree to which a document to be classified implicates a hypothesis sentence, the evaluation result aggregation unit 204 may calculate a statistic calculated from the implication scores calculated for each of the multiple hypothesis sentences as the overall score. In this case, the evaluation result aggregation unit 204 determines the classification destination of the document based on the calculated overall score. Note that the statistic is a numerical value that summarizes the feature quantities of data obtained by applying a statistical algorithm.

[0044] This configuration provides the effect of obtaining statistically valid evaluation results in addition to the effects of the document classification device 1 according to exemplary embodiment 1. For example, the evaluation result aggregating unit 204 may calculate, as the overall score, the arithmetic mean, mode, median, maximum, or minimum of the implication scores calculated for each of the multiple hypothesis sentences.

[0045] As described above, the document classification device 2 according to this exemplary embodiment is configured to include a hypothesis sentence generation unit 202 that generates, for each candidate destination for a document, a plurality of hypothesis sentences that are sentences related to the candidate destination, an entailment evaluation unit 203 that performs a process for each of the plurality of hypothesis sentences to evaluate whether the document to be classified entails the hypothesis sentence, and an evaluation result aggregation unit 204 that determines the destination of the document to be classified based on the evaluation result for each of the plurality of hypothesis sentences by the entailment evaluation unit 203. This configuration has the effect of enabling highly accurate evaluation results of validity to be obtained stably.

[0046] The documents to be classified may be character strings that have some meaning, and there are no particular limitations on their content, format, language, etc. Furthermore, the source of the documents to be classified is also not particularly limited. For example, character strings extracted from minutes of meetings, survey results, posts on social networking services (SNS), etc. may be used as documents to be classified, or documents that represent speech content converted into text by speech recognition may be used as documents to be classified. Furthermore, text extracted from data sources such as various databases may be used as documents to be classified, or premise sentences generated from the extracted text may be used as documents to be classified.

[0047] (Generation Strategy) The generation strategy is information for generating a hypothesis sentence related to a classification candidate. The generation strategy may be a template of a hypothesis sentence into which a character string of the classification candidate is incorporated to generate a hypothesis sentence. This will be described with reference to FIG. 5. FIG. 5 is a diagram showing an example of a generation strategy.

[0048] The table shown in FIG. 5 includes generation strategies 1 to 3. Such information is stored in the generation strategy holding unit 211. Generation strategy 1 is text data "This is a sentence about l." A hypothesis sentence is generated by substituting the character string of a classification destination candidate into the part of "l" in this text data. The same applies to generation strategies 2 and 3. By preparing such generation strategies, the hypothesis sentence generation unit 202 can easily generate multiple hypothesis sentences related to the classification destination candidates.

[0049] The method for generating a hypothesis sentence is not limited to the above example. For example, the hypothesis sentence generation unit 202 may generate a hypothesis sentence using a document generation model that receives a character string as input and outputs a document related to the character string. As the document generation model, for example, an encoder-decoder model or the like can be applied. The encoder-decoder model applied here encodes input text data (for example, converts it into a vector) and decodes the converted data (returns it to text data), thereby outputting a hypothesis sentence related to the input text data.

[0050] (Language understanding model) The language understanding model 212 is a model constructed so that, when a pair of a hypothesis sentence and a document to be evaluated is input, it outputs an implication score, which is an index value indicating the degree to which the document implicates the hypothesis sentence. A method for generating the language understanding model 212 will be explained with reference to Fig. 6. Fig. 6 is a diagram showing the method for generating the language understanding model 212.

[0051] The language understanding model 212 may be a combination of a pre-trained language model that converts documents into vectors based on their context, and a language task model that classifies documents. In this case, the pre-trained language model converts the document to be classified and the hypothesis sentence into vectors, and the language task model calculates an implication score that indicates the degree to which the document to be classified entails the hypothesis sentence from these vectors.

[0052] To generate such a language understanding model 212, first generate a pre-trained language model 62 from a large amount of text data 61, as shown in Figure 6. A self-supervised learning method is preferably used to generate the pre-trained language model 62. This allows learning to convert documents into vectors that are in line with their context, without labeling the text data with correct answer data. For example, the vast amount of text data on the web can be used directly for learning.

[0053] Next, a language task model 65 for classifying vectors generated by the pre-trained language model 62 is generated using labeled training data 63. Specifically, the training data 63 may be a set of a document and a hypothesis sentence, where it is known whether the document entails the hypothesis sentence, and a label indicating whether the document in the set entails the hypothesis sentence is attached to the set. For example, Stanford Natural Language Inference (SNLI) or Cross-lingual Natural Language Inference (XNLI) may be used as the training data 63.

[0054] This makes it possible to generate a language understanding model 212 that outputs an output value that represents the degree to which the input document implicates the input hypothesis sentence, for example, as a numerical value between 1 and 0. Note that, as shown in Figure 6, instead of using the pre-trained language model 62 as is, it is also possible to use a pre-trained language model 64 that has been tuned using training data 63 to improve compatibility with the language task model 65.

[0055] (Evaluation taking reliability into account) The reliability calculation unit 205 calculates the reliability of the generation strategy of the hypothesis sentence. The reliability is a numerical value indicating the validity of the generation strategy. For example, the reliability calculation unit 205 may calculate the reliability by applying a crowdsourcing technique. More specifically, the reliability calculation unit 205 may calculate the reliability of each generation strategy from the accuracy of the results of classification using each generation strategy recorded in the generation strategy storage unit 211.

[0056] This will be explained with reference to Fig. 7. Fig. 7 is a diagram showing an example of a method for calculating the reliability of a generation strategy and a method for calculating an overall score using the reliability. Fig. 7 shows the results of generating hypothesis sentences related to classification destination l1 using generation strategies 1 and 2, respectively, classifying multiple documents to be classified using each generated hypothesis sentence, and examining the accuracy of the classification results.

[0057] The documents to be classified include documents x1 to x3 extracted from the meeting minutes data (minutes 1 to 3), respectively. As shown in the figure, when the hypothesis sentence related to classification destination l1 generated by generation strategy 1 was used, the classification result for document x1 was correct, but the classification results for documents x2 and x3 were incorrect. On the other hand, when the hypothesis sentence related to classification destination l1 generated by generation strategy 2 was used, the classification results for all documents x1 to x3 were correct.

[0058] If history information indicating the accuracy of such classification results for various classification destinations 1 is recorded, the reliability calculation unit 205 can use the history information to calculate the reliability of each generation strategy for each classification destination 1. Note that the accuracy of the classification results (implication scores) can be determined by the user.

[0059] For example, the reliability calculation unit 205 may calculate the accuracy rate of each generation strategy from the history information, and calculate the reliability according to the accuracy rate. For example, if the accuracy rates of generation strategies 1 and 2 are 30% and 70%, respectively, the reliability calculation unit 205 may set the reliability of generation strategies 1 and 2 to 0.3 and 0.7, respectively.

[0060] When the reliability calculation unit 205 calculates the reliability, the evaluation result aggregation unit 204 considers the results of the evaluation by the implication evaluation unit 203 according to the reliability set for each of the multiple generation strategies. For example, the evaluation result aggregation unit 204 may calculate an overall score by weighting the implication scores calculated by the implication evaluation unit 203 by the reliability. For example, suppose that the reliability of generation strategies 1 and 2 is 0.3 and 0.7, respectively, and the implication scores corresponding to generation strategies 1 and 2 are 0.5 and 0.9, respectively. In this case, the evaluation result aggregation unit 204 may calculate a value of 0.3×0.5+0.7×0.9=0.78 as the overall score.

[0061] As described above, the hypothesis sentence generation unit 202 may generate multiple hypothesis sentences using multiple different generation strategies. Then, the evaluation result aggregation unit 204 may determine a classification destination for the document to be classified based on the result of the evaluation by the implication evaluation unit 203 and the reliability set for each of the multiple generation strategies. This configuration achieves the effect of obtaining a reasonable evaluation result that reflects the reliability of the hypothesis sentence generation strategy, in addition to the effects achieved by the document classification device 1 according to the first exemplary embodiment. Note that the process of determining a classification destination based on the evaluation result and the reliability is, in other words, a process of determining a classification destination by taking the evaluation result into consideration in accordance with its reliability.

[0062] (Processing flow) The flow of the process (document classification method) executed by the document classification device 2 will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the flow of the process executed by the document classification device 2.

[0063] In S21, the data acquisition unit 201 receives input of a document to be classified and candidate classification destinations. Any text data can be used as the document to be classified. The number of candidate classification destinations input may be one or more. For example, the data acquisition unit 201 may receive input of a label set L including multiple candidate classification destination labels l as candidate classification destinations.

[0064] In S22, the hypothesis sentence generation unit 202 generates a plurality of hypothesis sentences for each of the classification destination candidates received as input in S21. To generate the hypothesis sentences, for example, a generation strategy recorded in the generation strategy holding unit 211 may be used. For example, if there are five labels l1 to l5 that are classification destination candidates and the generation strategies are three, 1 to 3, three hypothesis sentences are generated for each of the labels l1 to l5. In this case, a total of 15 hypothesis sentences are generated.

[0065] In S23, the entailment evaluation unit 203 evaluates whether the document to be classified, whose input was accepted in S21, entails the hypothesis sentence generated in S22. For example, the entailment evaluation unit 203 may calculate an entailment score by inputting a pair of a hypothesis sentence and a document to be classified into the language understanding model 212. This process is performed for each of the multiple hypothesis sentences generated in S22. For example, if there are a total of 15 hypothesis sentences generated in S22, an entailment score is calculated for each of these hypothesis sentences, and the total number of calculated entailment scores will be 15.

[0066] In S24, the evaluation result aggregating unit 204 aggregates the evaluation results of S23 and determines the classification destination of the document to be classified that was received as input in S21. More specifically, the evaluation result aggregating unit 204 determines the classification destination of the document to be classified based on the evaluation results of S23 for each of the multiple hypothesis sentences, i.e., the implication scores. The classification destination of the document to be classified may be represented, for example, by the above-mentioned overall score. This process is performed for each of the classification destination candidates that were received as input in S21. For example, if there are five classification destination candidate labels, l1 to l5, a total score is calculated for each label, so the total number of calculated overall scores is five. These overall scores indicate the classification destination into which the document to be classified should be classified.

[0067] If the reliability calculation unit 205 has calculated the reliability, in S24 the evaluation result aggregating unit 204 calculates the overall score by taking into account the evaluation results of S23 according to the reliability set for each of the multiple generation strategies. The reliability may be calculated at any timing before the calculation of the overall score.

[0068] In S25, the evaluation result aggregating unit 204 outputs the classification destination determined by the processing in S24 to the output unit 23. For example, if there are five labels l1 to l5 that are candidate classification destinations and an overall score has been calculated for each of these labels, the evaluation result aggregating unit 204 may output the label whose calculated overall score exceeds a threshold. This ends the processing in FIG. 8.

[0069] In S25, the evaluation result aggregating unit 204 may output a total score for each of the classification destination candidates. In this case, the user of the document classification device 2 can use the output total score to determine which of the classification destination candidates the text to be classified into, or whether to classify it into any of the classification destination candidates. Of course, the evaluation results and classification destinations do not necessarily need to be output, and the evaluation result aggregating unit 204 may store the calculated evaluation results and the determined classification destinations in, for example, the storage unit 21, and then end the process.

[0070] [Modification] The execution entity of each process described in the above embodiment may be any entity and is not limited to the above example. In other words, a document classification system having the same functions as the document classification device 2 can be constructed using multiple devices that can communicate with each other. For example, by distributing each block shown in FIG. 4 among multiple devices, a document classification system having the same functions as the document classification device 2 can be constructed.

[0071] [Software implementation example] Some or all of the functions of the document classification device 2 may be realized by hardware such as an integrated circuit (IC chip), or by software. In the latter case, the document classification device 2 is realized by, for example, a computer that executes instructions of a program (document classification program), which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in FIG. 9. The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the document classification device 2. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the document classification device 2.

[0072] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0073] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0074] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0075] [Appendix 1] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.

[0076] [Appendix 2] Some or all of the above-described embodiments can also be described as follows: However, the present invention is not limited to the following described aspects.

[0077] (Appendix 1) A document classification device comprising: a hypothesis sentence generation means for generating, for each of candidate classification destinations for a document, a plurality of hypothesis sentences which are sentences related to the candidate classification destination; an implication evaluation means for performing a process for evaluating, for each of the plurality of hypothesis sentences, whether the document entails the hypothesis sentence; and an evaluation result aggregation means for determining a classification destination for the document based on the evaluation result by the implication evaluation means for each of the plurality of hypothesis sentences.

[0078] (Appendix 2) The document classification device described in Appendix 1, wherein the implication evaluation means calculates an index value indicating the degree to which the document entails the hypothesis sentence using a language understanding model constructed by learning whether the document entails the hypothesis sentence, and the evaluation result aggregation means determines where to classify the document based on statistics calculated from the index values ​​calculated for each of the multiple hypothesis sentences.

[0079] (Appendix 3) 3. The document classification device according to claim 1, wherein the hypothesis sentence generation means generates a plurality of the hypothesis sentences using a plurality of different generation strategies, and the evaluation result aggregation means determines where to classify the document based on the result of the evaluation by the implication evaluation means and a reliability set for each of the plurality of generation strategies.

[0080] (Appendix 4) A document classification method comprising: at least one processor generating, for each candidate classification destination for a document, a plurality of hypothesis sentences that are sentences related to the candidate classification destination; performing a process for evaluating, for each of the plurality of hypothesis sentences, whether the document implies the hypothesis sentence; and determining a classification destination for the document based on a result of the evaluation for each of the plurality of hypothesis sentences.

[0081] (Appendix 5) A document classification program that causes a computer to function as: hypothesis sentence generation means that generates, for each of candidate classification destinations for a document, a plurality of hypothesis sentences that are sentences related to the candidate classification destination; implication evaluation means that performs a process for evaluating, for each of the plurality of hypothesis sentences, whether the document implies the hypothesis sentence; and evaluation result aggregation means that determines the classification destination of the document based on the result of evaluation by the implication evaluation means for each of the plurality of hypothesis sentences.

[0082] [Appendix 3] Some or all of the above-described embodiments can also be expressed as follows: A document classification device comprising at least one processor, the processor executing a hypothesis sentence generation process that generates, for each of candidate classification destinations for a document, a plurality of hypothesis sentences that are sentences related to the candidate classification destination, an entailment evaluation process that evaluates, for each of the plurality of hypothesis sentences, whether the document entails the hypothesis sentence, and an evaluation result collection process that determines a classification destination for the document based on the evaluation results of the entailment evaluation process for each of the plurality of hypothesis sentences.

[0083] The document classification device may further include a memory that stores a program for causing the processor to execute the hypothesis sentence generation process, the implication evaluation process, and the evaluation result aggregation process. The program may also be recorded on a computer-readable, non-transitory, tangible recording medium. [Explanation of symbols]

[0084] 1, 2 Document classification device 11, 202 Hypothesis sentence generation section 12, 203 Implication Evaluation Section 13, 204 Evaluation Results Collection Department

Claims

1. a hypothesis sentence generation means for generating, for each of the document classification candidate, a plurality of hypothesis sentences that are sentences related to the document classification candidate; an implication evaluation means for performing a process of evaluating whether the document entails the hypothesis sentence for each of the plurality of hypothesis sentences; an evaluation result aggregation means for determining a classification of the document based on the result of evaluation by the implication evaluation means for each of the plurality of hypothesis sentences; the hypothesis sentence generation means generates the plurality of hypothesis sentences using a plurality of generation strategies that are different from each other; The evaluation result aggregation means determines the classification destination of the document based on the result of the evaluation by the implication evaluation means and the reliability set for each of the plurality of generation strategies.

2. the implication evaluation means calculates an index value indicating the degree to which the document implies the hypothesis sentence using a language understanding model constructed by learning whether the document implies the hypothesis sentence; 2. The document classification device according to claim 1, wherein the evaluation result aggregating means determines the classification of the document based on statistics calculated from the index values ​​calculated for each of the plurality of hypothesis sentences.

3. At least one processor generating, for each of the document classification candidates, a plurality of hypothesis sentences that are sentences related to the document classification candidate; performing a process for evaluating whether the document entails the hypothesis sentence for each of the plurality of hypothesis sentences; determining a classification destination for the document based on the results of the evaluation of each of the plurality of hypothesis sentences; In generating the hypothesis sentences, a plurality of the hypothesis sentences are generated using a plurality of different generation strategies; The document classification method includes determining a classification destination for the document based on the result of the evaluation and a reliability level set for each of the plurality of generation strategies.

4. Computer, a hypothesis sentence generation means for generating, for each of the document classification candidate, a plurality of hypothesis sentences which are sentences related to the document classification candidate; an implication evaluation means for performing a process of evaluating whether the document entails the hypothesis sentence for each of the plurality of hypothesis sentences; and functioning as an evaluation result aggregation means for determining a classification of the document based on the result of evaluation by the implication evaluation means for each of the plurality of hypothesis sentences; the hypothesis sentence generation means generates the plurality of hypothesis sentences using a plurality of generation strategies that are different from each other; The evaluation result aggregation means determines the classification destination of the document based on the result of the evaluation by the implication evaluation means and the reliability set for each of the plurality of generation strategies.

Images