Document classification system, document classification method and document classification program

Abstract

To provide a document classification system which facilitates classification of a document for which it is difficult to classify by a stipulated rule, in documents including multiple item values to multiple items, into classes on the basis of the multiple item values.SOLUTION: There is provided a document classification system for classifying documents into classes, on the basis of multiple item values of documents, configured so that: a document classification part provided on a processing part, receives input of the multiple item values of the document, then uses a rule base engine, for calculating a rule classification result being a classification result of the document, from the multiple item values of the documents input, then uses a machine classification model for calculating a machine classification result being the classification result of the document, from the multiple item values of the document input, uses a classification accuracy prediction model for calculating, a rule classification accuracy prediction value from the multiple item values of the document input, then, outputs the calculated rule classification result, the machine classification result and the rule classification accuracy prediction value.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a document classification system, a document classification method, and a document classification program that classify documents including a plurality of item values ​​into classes based on the plurality of item values. [Background technology]

[0002] Information is classified using a computer. For example, Patent Document 1 describes a parameter determination device that determines whether the state of an object is good or bad from image data including an image of the object by two methods: a rule-based determination method and a determination method using a machine learning model. The parameter determination device described in Patent Document 1 has a parameter determination unit, a rule-based determination unit, and a machine learning model determination unit. The parameter setting unit calculates thresholds Rth and Mth, which are parameters, based on a range of preset numerical values. The rule-based determination unit uses the threshold Rth and image data calculated by the parameter setting unit to determine whether the object is good or bad by the rule-based determination method. The machine learning model determination unit uses the threshold Mth and image data calculated by the parameter setting unit to determine the state by the machine learning model.

[0003] The parameter determination device described in Patent Document 1 uses two parameters, a threshold value Rth and a threshold value Mth, instead of using one parameter for both the rule-based determination unit and the machine learning model. As a result, the parameter determination device described in Patent Document 1 can determine parameters such that the determination accuracy of the determination method that uses both the rule-based determination method and the determination method using the machine learning model is increased. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-135659 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology described in Patent Document 1 is not easily applicable to classifying information other than images, and is also not applicable to machine learning models that do not require parameter input or models of rule-based judgment methods.

[0006] For example, the technology described in Patent Document 1 cannot be used to classify document data, which includes multiple items and values ​​(item values) for each of the multiple items, into classes based on the multiple item values. Here, the document to be classified is, for example, a document that includes multiple items related to personal assets and item values ​​representing the status of the personal assets for each of the multiple items, which is created to classify individuals into classes related to tax exemption according to the personal assets. Also, for example, the document to be classified is a document that includes multiple items related to a company's finances and item values ​​representing the financial status for each of the multiple items, which is created to classify companies into classes related to taxes and subsidies according to the financial status.

[0007] If document classification results are calculated using multiple methods, such as a document classification method using a machine learning model and a document classification method using a rule-based model, it is not easy to determine which classification result calculated by which method is to be trusted as the correct classification result. In addition, it is not easy to correctly classify documents that include item values ​​that fall under exceptions to rules built into the rule-based model, even if the rule-based model is used. Therefore, there is a demand for a technology that makes it easy to correctly classify documents that cannot be correctly classified into classes using the rule-based model.

[0008] Therefore, an object of the present invention is to provide a document classification support system, a document classification method, and a document classification program that make it easy to classify documents that contain multiple item values ​​for multiple items and that are not easy to classify according to written rules into classes based on the multiple item values. [Means for solving the problem]

[0009] In order to achieve the above object, one aspect of the document classification system of the present invention is a document classification system that classifies a document including a plurality of item values ​​for a plurality of items into a class based on the plurality of item values ​​of the document, the document classification system including a memory unit and a processing unit, the memory unit including a rule-based engine that is a model representing a relationship between the plurality of item values ​​of the document and the class, the rule-based engine being created based on a classification rule that classifies the document into the class based on the plurality of item values ​​of the document, a classification accuracy prediction model that is a machine learning model representing a relationship between the plurality of item values ​​of the document and a rule classification accuracy prediction value that indicates whether the document can be correctly classified using the rule-based engine, and a classification accuracy prediction model that is a machine learning model representing a relationship between the plurality of item values ​​of the document and the class. and a machine classification model which is a machine learning model representing the relationship between a class and a class, and the processing unit includes a document classification unit which classifies the document into the class, and when the multiple item values ​​of the document are input, the document classification unit uses the rule-based engine to calculate a rule classification result which is a classification result of the document from the multiple item values ​​of the input document, uses the machine classification model to calculate a machine classification result which is a classification result of the document from the multiple item values ​​of the input document, and uses the classification accuracy prediction model to calculate the rule classification accuracy predicted value from the multiple item values ​​of the input document, and outputs the calculated rule classification result, machine classification result, and rule classification accuracy predicted value.

[0010] Moreover, one aspect of the document classification method of the present invention is a document classification method in a document classification system including a storage device and a processor, in which the processor classifies a document including a plurality of item values ​​for a plurality of items into a class based on the plurality of item values ​​of the document, the document classification method including: a rule-based engine which is a model representing a relationship between the plurality of item values ​​of the document and the class, the rule-based engine being created based on a classification rule for classifying the document into the class based on the plurality of item values ​​of the document; a classification accuracy prediction model which is a machine learning model representing a relationship between the plurality of item values ​​of the document and a rule classification accuracy prediction value which indicates whether the document can be correctly classified using the rule-based engine; The processor stores a machine classification model, which is a machine learning model that represents the relationship between the multiple item values ​​of the document and the class, and when the multiple item values ​​of the document are input, the processor uses the rule-based engine to calculate a rule classification result that is the classification result of the document from the multiple item values ​​of the input document, uses the machine classification model to calculate a machine classification result that is the classification result of the document from the multiple item values ​​of the input document, and uses the classification accuracy prediction model to calculate the rule classification accuracy predicted value from the multiple item values ​​of the input document, and outputs the calculated rule classification result, machine classification result, and rule classification accuracy predicted value.

[0011] Moreover, one aspect of the document classification program of the present invention is a document classification system comprising a storage device and a processor, wherein the processor classifies a document including a plurality of item values ​​for a plurality of items into a class based on the plurality of item values ​​of the document, the storage device comprising: a rule-based engine which is a model representing the relationship between the plurality of item values ​​of the document and the class, the model being created based on classification rules for classifying the document into the class based on the plurality of item values ​​of the document; a classification accuracy prediction model which is a machine learning model representing the relationship between the plurality of item values ​​of the document and a rule classification accuracy prediction value which indicates whether the document can be correctly classified using the rule-based engine; and a machine classification model, which is a machine learning model that represents the relationship between the item values ​​of the document and the class, and when the multiple item values ​​of the document are input, the processor is caused to use the rule-based engine to calculate a rule classification result, which is the classification result of the document, from the multiple item values ​​of the input document, use the machine classification model to calculate a machine classification result, which is the classification result of the document, from the multiple item values ​​of the input document, and use the classification accuracy prediction model to calculate the rule classification accuracy predicted value from the multiple item values ​​of the input document, and output the calculated rule classification result, machine classification result, and rule classification accuracy predicted value. Effect of the Invention

[0012] According to a representative embodiment of the present invention, among documents including multiple item values ​​for multiple items, documents that are difficult to classify according to written rules can be easily classified into classes based on the multiple item values. Problems, configurations, and effects other than those described above will become apparent from the description of the following embodiments. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram showing an example of an outline of the configuration of a document classification system according to an embodiment. [Diagram 2]FIG. 2 is a diagram for explaining a document classification method of the document classification system. [Diagram 3] FIG. 3 is a diagram illustrating an example of a functional block diagram of the document classification system according to the embodiment. [Figure 4] FIG. 4 is a block diagram illustrating an example of a hardware configuration of the document classification support device according to the embodiment. [Diagram 5] FIG. 5 is a diagram for explaining data relating to document classification. [Figure 6] FIG. 6 is a diagram illustrating an example of document data store 232. [Figure 7] FIG. 7 is a diagram showing an example of the training data information table 235. As shown in FIG. [Figure 8] FIG. 8 is a diagram showing an example of the test data information table 236. As shown in FIG. [Figure 9] FIG. 9 is a flowchart illustrating an example of a classification accuracy prediction model generation process. [Figure 10] FIG. 10 is a diagram showing an example of the classification result display screen. [Figure 11] FIG. 11 is a diagram showing an example of the classification basis display screen. [Figure 12A] FIG. 12A is a tree diagram showing, in the form of a tree, the rules for classifying documents built into the rule-based engine before the update. [Figure 12B] FIG. 12B is an example of a tree diagram showing that the judgment conditions shown in the additional classification rules are incorporated into the rule-based engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention should not be interpreted as being limited to the description of the embodiment shown below. It will be easily understood by those skilled in the art that the specific configuration can be changed without departing from the concept or purpose of the present invention.

[0015] In the configurations of the invention described below, the same or similar configurations or functions are given the same reference numerals, and duplicated explanations are omitted.

[0016] In this specification, the terms "first," "second," "third," and the like are used to identify components and do not necessarily limit the number or order.

[0017] In this specification, as an example of various information, the term "XX table" may be used, but it may also be expressed as a data structure such as an "XX list" or "XX queue." Furthermore, the "XX table" may also be expressed as "XX information." When explaining identification information, the terms "identification information," "identifier," "name," "ID," "number," and the like are used, but these are interchangeable. EXAMPLES

[0018] <<System configuration and processing overview>> <System Overview> Fig. 1 is a diagram showing an example of an outline of the configuration of a document classification system 1 according to an embodiment. As shown in Fig. 1, the document classification system 1 includes a document classification support device 100 and a plurality of document classification terminals 200. The document classification support device 100 and the document classification terminals 200 are connected via a network NW.

[0019] The document classification support device 100 manages models (a rule-based engine RE, a machine classification model MM, and a classification accuracy prediction model AP, which will be described later) used to classify documents into classes, as well as information on the documents and the document classification results. The document classification support device 100 is primarily used by the model administrator. The document classification support device 100 can also be used to classify documents, similar to the document classification terminal 200.

[0020] An example will be described below. The document to be classified is a document that includes multiple items related to a situation and values ​​(item values) that indicate the situation for the multiple items. A class is a classification related to a situation. The document to be classified is, for example, a document such as an application document. The document to be classified is, for example, a document that includes multiple items related to an individual's assets and item values ​​that indicate the status of the individual's assets for each of the multiple items, which are created to classify an individual into classes related to tax exemption according to the individual's assets. Then, a person in charge of calculating an individual's taxes may use the document classification system 1 to classify documents related to the individual's assets. Also, an individual may use the document classification system 1 to calculate an exemption tax.

[0021] Also, for example, the documents to be classified are documents that include multiple items related to an individual's health condition and item values ​​that indicate the individual's health condition for each of the multiple items, which are created to classify individuals into classes related to medical insurance according to the individual's health condition. Then, for example, a person in charge of calculating the cost of medical insurance for an individual may use the document classification system 1 to classify documents related to the individual's health condition into classes and calculate the cost of medical insurance for an individual.

[0022] Also, for example, the documents to be classified are documents that contain multiple items related to the company's finances and item values ​​that indicate the financial status of each of the multiple items, which are created to classify the company into classes related to taxes and subsidies according to the financial status of the company. Then, for example, a person in charge of calculating taxes and subsidies for a company may use the document classification system 1 to classify documents related to the company's finances and calculate the taxes and subsidies for the company.

[0023] The document classification terminal 200 is used to classify documents using models managed by the document classification support device 100. The document classification terminal 200 classifies documents into classes using the models and can display the document classification results. The document classification terminal 200 transmits the document classification results to the document classification support device 100. The document classification results are stored in the document classification support device 100. The document classification terminal 200 is primarily used by document classifiers. A document classifier is a person in charge of classifying documents.

[0024] The general terminal 500 can be used to access the document classification support device 100 and create a document. The general terminal 500 can be used by a document applicant. The document applicant can use the general terminal 500 to access the document classification support device 100 and create digital data of the document, and submit (save) the created digital data of the document to the document classification support device 100.

[0025] Also, a document applicant can submit a paper document to the administrator. Then, the administrator can use the document classification support device 100 to scan the submitted paper document and generate digital data of the document. Then, the administrator can store the generated digital data of the document in the document classification support device 100. The administrator can also use the document classification support device 100 to classify documents (digital data of the document).

[0026] Furthermore, a document applicant can submit a paper document to a document classifier. The document classifier can then use the document classification terminal 200 to scan the submitted paper document to generate digital data of the document, and store the generated digital data of the document in the document classification support device 100.

[0027] FIG. 2 is a diagram explaining a document classification method of the document classification system 1. As shown in FIG. 2, the document classification system 1 classifies documents based on multiple item values ​​of the documents using each of the rule-based engine RE, the machine classification model MM, and the classification accuracy prediction model AP, and displays the classification results. Here, the document classification system 1 classifies documents into classes using the rule-based engine RE. The document classification system 1 classifies documents into classes using the machine classification model MM. The document classification system 1 classifies documents using the classification accuracy prediction model AP to determine whether the documents can be correctly classified using the rule-based engine RE.

[0028] The rule-based engine RE is created based on written rules (examination criteria, etc.) for classifying documents. Therefore, when the document classification system 1 uses the rule-based engine RE to classify documents, the document is classified using the written rules, which are explicit knowledge that are incorporated into the rule-based engine RE.

[0029] On the other hand, the machine classification model MM is a machine learning model. Therefore, when the document classification system 1 classifies documents using the machine classification model MM, the document is classified using the explicit knowledge and tacit knowledge rules learned by the machine classification model MM.

[0030] Based on the document classification result calculated using the classification accuracy prediction model AP (whether the document can be correctly classified using the rule-based engine RE) displayed by the document classification system 1, the document classifier can consider which value is more useful, the classification result calculated using the rule-based engine RE or the classification result calculated using the machine classification model MM. In other words, if the document classification result calculated using the classification accuracy prediction model AP indicates that the document can be correctly classified using the rule-based engine RE, the document classifier can classify the document into a class represented by the classification result obtained by the document classification system 1 classifying the document using the rule-based engine RE. Also, if the document classification result calculated using the classification accuracy prediction model AP indicates that the document cannot be correctly classified using the rule-based engine RE, the document classifier can classify the document into a class represented by the classification result obtained by the document classification system 1 classifying the document using the machine classification model MM.

[0031] In this way, the document classifier can classify documents based on a highly reliable classification result between the classification result according to the written rules, which is explicit knowledge (classification result using the rule-based engine RE) and the classification result according to the classification method learned by the machine classification model MM, which is explicit knowledge and tacit knowledge. This makes it easier for the document classifier to classify documents that are not easy to classify according to written rules into classes. Therefore, the document classification system 1 makes it easier to classify documents that include multiple item values ​​for multiple items and that are not easy to classify according to written rules into classes based on the multiple item values.

[0032] Furthermore, the document classification system 1 can be used to incorporate into the rule-based engine RE the rules for classifying documents that are incorporated into the machine classification model MM but are incorporated into the rule-based engine RE, thereby making it possible to improve the accuracy of document classification by the rule-based engine RE by explicitly writing down the tacit rules that are incorporated into the machine classification model MM and incorporating them into the rule-based engine RE.

[0033] Fig. 3 is a diagram showing an example of a functional block diagram of the document classification system 1 of the embodiment. As shown in Fig. 3, the document classification support device 100 has a storage unit 11, a processing unit 12, a display unit 13, and a scanner 241. The processing unit 12 includes a document information acquisition unit 121, a model creation unit 122, a model evaluation unit 123, a model update unit 124, a document classification unit 125, and a document creation unit 126.

[0034] The document sorting terminal 200 has a memory unit 11 , a processing unit 12 including a document information acquisition unit 121 and a document sorting unit 125 , a display unit 13 , and a scanner 241 .

[0035] The storage unit 11 stores documents, a model for classifying documents, information for updating the model, and the like.

[0036] Document information acquisition section 121 acquires documents (digital data of documents) from memory section 11 and scanner 241, and outputs the acquired documents to document classification section 125. Document information acquisition section 121 also stores document information in document data store 232 and document information table 234 as appropriate.

[0037] The model creation unit 122 creates an initial model (a model created first). The model evaluation unit 123 calculates a value for evaluating a model including the initial model (the accuracy of the classification result when a document is classified using the model). When information on the classification results of one or more documents is accumulated, the model update unit 124 uses the accumulated documents and the document classification results as learning data to train the model. By training the model in this manner, the model is updated. Here, the model learns the learning data through training. Note that the model update unit 124 may update the model using one document and one classification result.

[0038] The document creation unit 126 accepts input of multiple item values ​​of a document from the general terminal 500 that has accessed the document classification support device 100, and transmits information of a document creation screen (not shown) used to create a document (digital data of the document) to the general terminal 500. The general terminal 500 displays the document creation screen and accepts input of multiple item values ​​of the document. When multiple item values ​​of the document are input, the general terminal 500 transmits the input multiple item values ​​of the document to the document classification support device 100. The document creation unit 126 creates a document (digital data of the document) from the multiple item values ​​received from the general terminal 500, and stores information about the created document in the document data store 232 and the document information table 234.

[0039] Both the document classification support device 100 and the document classification terminal 200 have a storage unit 11, a document information acquisition unit 121, a document classification unit 125, a display unit 13, and a scanner 241. The scanner 241 reads a paper document, generates digital data of the document (also referred to as a "document"), and transmits the generated digital data of the document to the document information acquisition unit 121. As described above, the document information acquisition unit 121 acquires a document from the document data store 232 of the storage unit 11 and the scanner 241, and outputs the acquired document to the document classification unit 125. The document classification unit 125 classifies the document acquired from the document information acquisition unit 121 using a model, and outputs the document classification result to the display unit 13. The display unit 13 displays the document classification result. In addition, the model evaluation unit 123 causes the storage unit 11 to store the document class information input by the document classifier. Both the document sorting assistance device 100 and the document sorting terminal 200 can be used for sorting documents in this manner.

[0040] <Hardware Configuration of Document Classification Support Device 100> Fig. 4 is a block diagram showing an example of the hardware configuration of the document classification support device 100. As shown in Fig. 4, the document classification support device 100 has a processor 21, a main memory device 22, a sub-memory device 23, an input device 24, an output device 25, a network I / F 26, and a bus 27 connecting these. The document classification support device 100 can be realized by a general information processing device such as a PC or a server computer. The document classification support device 100 may be built on a server in a cloud environment.

[0041] The processor 21 reads data and programs stored in the sub-storage device 23 into the main storage device 22 and executes processing determined by the programs. The processing unit 12 is the processor 21. The document information acquisition unit 121 described above with reference to FIG. 3 is realized by the processor 21 reading document information acquisition program 121a stored in the sub-storage device 23, which will be described later, into the main storage device 22 and executing it. In this specification, when a process is described in a sentence with document information acquisition unit 121, which is a functional unit, as the subject, it indicates that the processor 21 is executing document information acquisition program 121a that realizes the functional unit. Similarly, the model creation unit 122, the model evaluation unit 123, the model update unit 124, the document classification unit 125, and the document creation unit 126 are realized by the processor 21 reading model creation program 122a, model evaluation program 123a, model update program 124a, document classification program 125a, and document creation program 126a stored in the sub-storage device 23 into the main storage device 22 and executing them, respectively. In this specification, when processing is described in a sentence with each of the functional units, namely model creation unit 122, model evaluation unit 123, model update unit 124, document classification unit 125 and document creation unit 126, as the subject, it indicates that processor 21 is executing model creation program 122a, model evaluation program 123a, model update program 124a, document classification program 125a and document creation program 126a, respectively, which realize the corresponding functional units.

[0042] The main memory device 22 includes a volatile memory element such as a RAM, and stores the programs executed by the processor 21 and data.

[0043] The secondary storage device 23 is a device having a non-volatile storage element such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores programs, data, etc. The secondary storage device 23 stores a reference rule database 231, a document data store 232, a model store 233, a document information table 234, a training data information table 235, and a test data information table 236. The secondary storage device 23 also has installed therein a document information acquisition program 121a, a model creation program 122a, a model evaluation program 123a, a model update program 124a, a document classification program 125a, and a document creation program 126a.

[0044] The standard rule database 231 stores information on rules for classifying documents (for example, legal examination standards).

[0045] The document data store 232 stores information associating documents with their names.

[0046] The model store 233 stores information on various models.

[0047] The document information table 234 stores information about the documents stored in the document data store 232 .

[0048] The training data information table 235 stores information about the training data used to train the model.

[0049] The test data information table 236 stores information about the test data used to evaluate the model.

[0050] The input device 24 is one or more devices such as a keyboard or a mouse that accepts user operations and acquires information input by the user operations. The input device 24 includes the scanner 241 described above with reference to FIG.

[0051] The output device 25 is a device that outputs information, such as a display, and presents information to a user by displaying it on a screen, for example. The document classification support device 100 may include a touch panel that serves as both the input device 24 and the output device 25.

[0052] The network I / F 26 is an interface capable of transmitting and receiving data via the network NW with devices such as the document sorting terminal 200 and the general terminal 500. Using the network I / F 26, the document sorting support device 100 can transmit and receive data with devices such as the document sorting terminal 200 and the general terminal 500 that are connected to the network NW.

[0053] 2 is the secondary storage device 23. The display unit 13 is one of the output devices 25, such as a monitor or a touch panel.

[0054] The document classification terminal 200 and the general terminal 500 can be realized using the same hardware resources as the document classification support device 100 .

[0055] <Various data structures> FIG. 5 is a diagram for explaining data relating to document classification. A document includes a plurality of items and values ​​(item values) for each of the plurality of items. The number of item values ​​included in one document is, for example, 10. Furthermore, there are five types of classes to be classified, Class A to Class E. The classes may be grades or ranks that have different degrees of magnitude (for example, Grade A to Grade E). Furthermore, the classes may be classes of different types that do not take into account differences in degree. The item values ​​are, for example, numbers, sentences, Yes / No, and option values. There is no limit to the type of item value. Furthermore, there is no limit to the number of item values ​​included in one document. Furthermore, there is no limit to the type of classes to be classified.

[0056] The rule-based engine RE is a model that represents the relationship between a document's multiple item values ​​and its class, created based on the rules for classifying documents (such as the screening criteria stored in the reference rule database 231). In the rule-based engine RE, the multiple item values ​​contained in a document are the values ​​of explanatory variables, and the class is the value of the objective variable. The document classification unit 125 of the document classification support device 100 and the document classification unit 125 of the document classification terminal 200 use the rule-based engine RE to calculate the document's class from the document's multiple item values, and this class is referred to as the "rule classification result vre."

[0057] The rule classification correctness judgment value vrej is a numerical value indicating whether the rule classification result vre is a correct value (correct class). The rule classification correctness judgment value vrej indicates whether a document can be correctly classified using the rule-based engine RE (whether the rule classification result vre is correct or not). The value of the rule classification correctness judgment value vrej is either "0", which indicates that the rule classification result vre is a correct value (e.g., the value of the correct label), or "X", which indicates that the rule classification result vre is an incorrect value (e.g., a value different from the correct label).

[0058] The rate at which the rule classification result vre is a correct value (the accuracy of the rule classification result vre) is defined as the rule classification result accuracy vrea. As will be described later, the model evaluation unit 123 of the document classification support device 100 calculates the rule classification result accuracy vrea by using test data.

[0059] The classification accuracy prediction model AP is a machine learning model that represents the relationship between multiple item values ​​of a document and a rule classification correctness judgment value vrej that indicates whether the document can be correctly classified using the rule-based engine RE (whether the rule classification result vre is correct or not). A machine learning rule-based model can be used as the classification accuracy prediction model AP. In the classification accuracy prediction model AP, the multiple item values ​​included in the document are the values ​​of explanatory variables, and the rule classification correctness judgment value vrej is the value of the objective variable.

[0060] The value indicating whether the document classification unit 125 of the document classification support device 100 and the document classification unit 125 of the document classification terminal 200 can correctly classify a document using the rule-based engine RE, calculated from multiple item values ​​contained in the document using the classification accuracy prediction model AP, is defined as the "rule classification accuracy prediction value vap." The value of the rule classification accuracy prediction value vap is either "0," which indicates that the document can be correctly classified using the rule-based engine RE, calculated from multiple item values ​​contained in the document using the classification accuracy prediction model AP, or "X," which indicates that the document cannot be correctly classified.

[0061] A value indicating whether the rule classification accuracy predicted value vap is a correct value (correct answer) is referred to as the rule classification accuracy predicted value correctness judgment value vapj. The value of the rule classification accuracy predicted value correctness judgment value vapj is either "0" indicating that the rule classification accuracy predicted value vap is a correct value (correct answer: the rule classification accuracy predicted value vrej and the rule classification accuracy predicted value vap are the same value), or "X" indicating that the rule classification accuracy predicted value vap is a value that is not a correct value (correct answer) (the rule classification accuracy predicted value vrej and the rule classification accuracy predicted value vap are different).

[0062] The rate at which the rule classification accuracy predicted value vap is correct (the accuracy of the rule classification accuracy predicted value vap) is defined as the rule classification accuracy predicted value accuracy vapa. As will be described later, the model evaluation unit 123 of the document classification support device 100 calculates the rule classification accuracy predicted value accuracy vapa using test data.

[0063] The machine classification model MM is a machine learning decision tree model that represents the relationship between the values ​​of multiple items of a document and a class. A machine learning model other than a decision tree model may be used as the machine classification model MM. In the machine classification model MM, the multiple item values ​​contained in a document are the values ​​of explanatory variables, and the class is the value of a target variable. The document classification unit 125 of the document classification support device 100 and the document classification unit 125 of the document classification terminal 200 use the machine classification model MM to calculate the class of the document from the multiple item values ​​of the document, and this class is referred to as the "machine classification result vmm."

[0064] The value indicating whether the machine classification result vmm is a correct value (correct class) is referred to as the machine classification correctness judgment value vmmj. The machine classification correctness judgment value vmmj indicates whether a document can be correctly classified using the machine classification model MM (whether the machine classification result vmm is correct or not). The value of the machine classification correctness judgment value vmmj is either "0", which indicates that the machine classification result vmm is a correct value (correct class; for example, correct answer), or "X", which indicates that the machine classification result vmm is an incorrect value.

[0065] The rate at which the machine classification result vmm is correct (the accuracy of the machine classification result vmm) is defined as the machine classification result accuracy vmma. As will be described later, the model evaluation unit 123 of the document classification support device 100 calculates the machine classification result accuracy vmma by using test data.

[0066] The document classification assistance device 100 uses learning data to train and evaluate the rule-based engine RE, the classification accuracy prediction model AP, and the machine classification model MM. The learning data includes training data and test data. The training data is used to train the model. The test data is used to evaluate the model, for example, to calculate the accuracy of the classification result calculated using the model.

[0067] The training data includes a plurality of pairs of a training document (referred to as a "training document") and a correct label which is a class of the training document. A record including information associating the training document name of the training data with the correct label is stored in the training data information table 235.

[0068] The test data includes a plurality of pairs of test documents (referred to as "test documents") and correct labels which are classes of the test documents. Records including information associating the test document names of the test data with the correct labels are stored in the test data information table 236. The training documents and the test documents are stored in the document data store 232 of the document classification assistance device 100.

[0069] The training data and the test data include rule-classifiable set data and rule-incompatible set data. The rule-classifiable set data is data of pairs of multiple item values ​​of a document that can be correctly classified using the rule-based engine RE and a correct label (correct class). The rule-incompatible set data is data of pairs of multiple item values ​​of a document that cannot be correctly classified even using the rule-based engine RE and a correct label (correct class).

[0070] Fig. 6 is a diagram showing an example of the data configuration of document data store 232. Document data store 232 shown in Fig. 6 has fields of document type 601, document name 602, and class 603. The value of type 601 is the type of document (for example, medical examination document, class type determined by the document). When an administrator or document classifier inputs a document class into document classification support device 100 or document classification terminal 200, the class is saved as the value of class 603. On the other hand, when an administrator or document classifier does not input a document class, no class is saved as the value of class 603.

[0071] Fig. 7 is a diagram showing an example of the data configuration of the training data information table 235. The training data information table 235 shown in Fig. 7 has fields including type 701, training document name 702, correct answer label 703, training use date 704, rule classification result vLre 705, and rule classification correct / incorrect judgment value vLrej 706. The training data information table 235 stores information about the training data.

[0072] The value of type 701 is the type of document (e.g., a medical examination document, a class type determined by the document). The value of correct answer label 703 is the class for training document name 702 stored in document data store 232. The value of training use date 704 stores the date on which the document with training document name 702 (training document) was used to train the model. The rule classification result vre for the training document is referred to as the "rule classification result vLre". Also, the rule classification correctness judgment value vrej for the training document is referred to as the "rule classification correctness judgment value vLrej".

[0073] As will be described below, the rule classification result vLre 705 and the rule classification correctness judgment value vLrej 706 are calculated in the course of model training and are stored in the training data information table 235 .

[0074] Fig. 8 is a diagram showing an example of the data configuration of the test data information table 236. The test data information table 236 shown in Fig. 8 has fields including type 801, test document name 802, correct answer label 803, test use date 804, rule classification result vTre 805, rule classification correctness judgment value vTrej 806, rule classification accuracy prediction value vTap 807, rule classification accuracy prediction value correctness judgment value vTapj 808, machine classification result vTmm 809, and machine classification correctness judgment value vTmmj 810. The test data information table 236 stores information about the test data.

[0075] The value of type 801 is the type of document (e.g., medical examination document, type of class determined by document). The value of correct answer label 803 is the class for test document name 802 stored in document data store 232. The value of test use date 804 stores the date when the document with test document name 802 was used to evaluate (test) the model. The rule classification result vre for the test document is referred to as "rule classification result vTre". The rule classification correctness judgment value vrej for the test document is referred to as "rule classification correctness judgment value vTrej". The rule classification accuracy prediction value vap for the test document is referred to as "rule classification accuracy prediction value vTap". The rule classification accuracy prediction value correctness judgment value vapj for the test document is referred to as "rule classification accuracy prediction value correctness judgment value vTapj". The machine classification result vmm for the test document is referred to as "machine classification result vTmm". The machine classification correctness judgment value vmmj for the test document is referred to as the "machine classification correctness judgment value vTmmj."

[0076] As described below, the values ​​of the rule classification result vTre805, the rule classification correctness judgment value vTrej806, the rule classification accuracy prediction value vTap807, the rule classification accuracy prediction value correctness judgment value vTapj808, the machine classification result vTmm809, and the machine classification correctness judgment value vTmmj810 are calculated in the process of evaluating the model (calculating the accuracy) and are stored in the test data information table 236.

[0077] <Processing Procedure> The following (1) to (4) will be described as processes performed by the document classification system 1: (1) Generation of an initial model, (2) Evaluation of the initial model, (3) Classification of documents using the model, and (4) Updating the model and evaluation of the updated model.

[0078] (1) Generation of the initial model An initial model is a first model used to classify documents. The initial model is generated by the model creation unit 122. The initial model of the rule-based engine RE is referred to as "rule-based engine RE1." The initial model of the classification accuracy prediction model AP is referred to as "classification accuracy prediction model AP1." The initial model of the machine classification model MM is referred to as "machine classification model MM1."

[0079] The following describes (1A) the generation of the rule-based engine RE1, (1B) the generation of the machine classification model MM1, and (1C) the generation of the classification accuracy prediction model AP1. (1A) The rule-based engine RE1 is generated using information on a classification method (e.g., screening criteria, etc.) for classifying documents into classes, which is stored in the reference rule database 231. (1B) The machine classification model MM1 and (1C) the classification accuracy prediction model AP1 are generated by training using training data.

[0080] (1A) Generation of rule-based engine RE1. (1A1) The document classification support device 100 obtains information on a classification method (e.g., screening criteria, etc.) for classifying documents into classes based on multiple item values ​​from the standard rule database 231, and displays the information on the classification method for classifying documents into classes on a display (output device 25). (1A2) An administrator or the like generates and inputs classification rules for classifying documents into classes based on multiple item values. (1A3) The model creation unit 122 converts the input classification rule into a classification command including a conditional branch such as If-Then, and generates a rule-based engine RE1 from the converted command. Then, the model creation unit 122 stores the rule-based engine RE1 in the model store 233.

[0081] The operations (1A1) and (1A2) may be performed by the model creation unit 122 by incorporating a program using a known technology capable of performing part or all of the operations (1A1) and (1A2) into the model creation program 122a. The model creation unit 122 may perform text mining processing or the like on information on a classification method for classifying documents into classes (e.g., screening criteria, etc.) stored in the reference rule database 231, to create classification rules, convert the classification rules into classification commands including conditional branches such as If-Then, and generate the rule-based engine RE1.

[0082] (1B) Generation of the machine classification model MM1. (1B1) The model creation unit 122 acquires, from the model store 233, a learning machine classification prototype model pMM that represents the relationship between a plurality of item values ​​of a document and classes. (1B2) The model creation unit 122 trains the acquired learning machine classification prototype model pMM using the training data, and sets the trained learning machine classification prototype model pMM as a machine classification model MM1. Here, the model creation unit 122 refers to the training data information table 235 and acquires a list of training document names of all records stored in the training data information table 235. The model creation unit 122 acquires a document associated with each training document name included in the list of training document names from the document data store 232, sets the document as a training document, and acquires a correct label associated with the training document name in the training data information table 235, sets the correct label. The model creation unit 122 trains the learning machine classification prototype model pMM using pairs of all acquired training documents and correct labels, and sets the trained learning machine classification prototype model pMM as a machine classification model MM1. Then, the model creation unit 122 stores the machine classification model MM1 in the model store 233. Here, by training the learning machine classification prototype model pMM, a machine classification model MM1 is generated in which the learning machine classification prototype model pMM is trained using pairs of all training documents and correct labels.

[0083] The training data includes rule-classifiable group data and rule-incompatible group data. Rule-classifiable group data is data of pairs of multiple item values ​​of a document that can be correctly classified using the rule-based engine RE and correct labels (correct classes). Rule-incompatible group data is data of pairs of multiple item values ​​of a document that cannot be correctly classified even using the rule-based engine RE and correct labels (correct classes).

[0084] (1C) Generation of classification accuracy prediction model AP1. 9 is a flowchart showing an example of a classification accuracy prediction model generation process for generating the classification accuracy prediction model AP1 of the embodiment. The model creation unit 122 of the document classification assistance device 100 executes the classification accuracy prediction model generation process shown in FIG.

[0085] The model creating unit 122 acquires, from the model store 233, a classification accuracy prediction prototype model pAP that represents the relationship between a plurality of item values ​​of a document and a class (step S110).

[0086] Next, the model creation unit 122 starts a loop process to calculate a rule classification correctness judgment value vLrej that indicates whether or not a training document can be correctly classified into a class using the rule-based engine RE, which is used for training the classification accuracy prediction prototype model pAP (step S120). The process shown from step S120 at the start of the loop to step S160 at the end of the loop is repeatedly performed for each training document name 702 of a record (record with a blank value for the training use date 704) in the training data information table 235. In step S120, the model creation unit 122 selects one training document name from among the unprocessed training document names stored in the training data information table 235.

[0087] Next, the model creating unit 122 acquires the training document associated with the training document name from the document data store 232 (step S130).

[0088] Next, the model creation unit 122 uses the rule-based engine RE1 to calculate the rule classification result vLre, which is the classification result by classifying the training document obtained in step S130, and stores the calculated rule classification result vLre in the training data information table 235 in association with the training document name (step S140).

[0089] Next, the model creation unit 122 calculates a rule classification correctness judgment value vLrej indicating whether the rule classification result vLre is correct or not, and stores the calculated rule classification correctness judgment value vLrej in the training data information table 235 (step S150). The model creation unit 122 determines whether the rule classification result vLre calculated in step S140 is the same as the correct label stored in association with the training document name in the training data information table 235. When the rule classification result vLre is the same as the correct label, the model creation unit 122 calculates the value "O" indicating that the rule classification result vLre is a correct value as the rule classification correctness judgment value vLrej (rule classification correctness judgment value vLrej=0). On the other hand, when the rule classification result vLre differs from the correct label, the model creation unit 122 calculates a value "X" indicating that the rule classification result vLre is an incorrect value as the rule classification correctness judgment value vLrej (rule classification correctness judgment value vLrej=X). Then, the model creation unit 122 stores the calculated rule classification correctness judgment value vLrej in the training data information table 235 in association with the training document name.

[0090] Next, the model creation unit 122 determines whether or not the processing has been completed for all training document names stored in the training data information table 235 (step S160). If the model creation unit 122 determines that the processing has not been completed for the training document names 702 of all records (records in which the value of the training use date 704 in the training data information table 235 is blank), the model creation unit 122 returns to step S120 and continues the processing. On the other hand, if the model creation unit 122 determines that the processing has been completed for the training document names 702 of all records, the model creation unit 122 executes the processing of step S170.

[0091] Next, the model creation unit 122 trains the classification accuracy prediction prototype model pAP using multiple pairs of the training documents acquired in step S130 and the rule classification correct / incorrect judgment value vLrej calculated in step S150 as training data, generates the classification accuracy prediction model AP1, and ends the process (step S170). The model creation unit 122 stores the generated classification accuracy prediction model AP1 in the model store 233. The model creation unit 122 stores information on the current day in the training use date 704 of all records (records in which the value of the training use date 704 in the training data information table 235 is blank). Here, by training the classification accuracy prediction prototype model pAP, the classification accuracy prediction model AP1 is generated by learning the classification accuracy prediction prototype model pAP using pairs of all training documents and correct answer labels.

[0092] (2) Evaluation of the initial model The model evaluation unit 123 uses the rule-based engine RE1, the machine classification model MM1, and the classification accuracy prediction model AP1 to calculate the classification result of the test documents of the test data, and further calculates whether the classification result is correct or not, and further calculates the accuracy of the classification result (referred to as "classification accuracy"). Records stored in the test data information table 236 that do not store a value for the test use date 804 are referred to as "records to be evaluated". The model update unit 124 extracts the records to be evaluated from the test data information table 236, and stores them in the main memory device 22.

[0093] (2A) Evaluation of the rule-based engine RE1. The model evaluation unit 123, which will be described in detail below, performs the following processes (2A1) and (2A2) for each test document name 802 of the evaluation target record. Then, the model update unit 124 performs the process (2A3). (2A1) Using the rule-based engine RE1, calculates a rule classification result vTre, which is the classification result of the test document associated with the test document name 802. (2A2) Calculates a rule classification correctness judgment value vTrej indicating whether the rule classification result vTre is a correct value (correct answer). (2A3) Using the rule classification correctness judgment value vTrej, calculates the proportion of the rule classification result vTre that is a correct value (the accuracy of the rule classification result vre) as a rule classification result accuracy vrea.

[0094] (2A1) Calculation of the rule classification result vTre using the rule-based engine RE1. The model evaluation unit 123 acquires a test document associated with the test document name of the evaluation target record from the document data store 232. The model evaluation unit 123 classifies the acquired test document using the rule-based engine RE1, and calculates a rule classification result vTre, which is the classification result. The model evaluation unit 123 stores the calculated rule classification result vTre in the test data information table 236 in association with the test document name.

[0095] (2A2) Calculation of the rule classification correctness judgment value vTrej. The model evaluation unit 123 calculates a rule classification correctness judgment value vTrej indicating whether the rule classification result vTre is a correct value (correct answer). When the rule classification result vTre is a correct value (correct answer), the rule classification result vTre is the same value (class) as the correct answer label of the record to be evaluated. Therefore, when the value of the rule classification result vTre is the same as the correct answer label, the model evaluation unit 123 sets the value of the rule classification correctness judgment value vTrej to a value "0" indicating that the rule classification result vTre is correct (rule classification correctness judgment value vTrej = 0). On the other hand, when the value of the rule classification result vTre is different from the correct answer label, the model evaluation unit 123 sets the value of the rule classification correctness judgment value vTrej to a value "X" indicating that the rule classification result vTre is incorrect (rule classification correctness judgment value vTrej = X). Then, the model evaluation unit 123 stores the calculated rule classification true / false judgment value vTrej (0 or X) in the test data information table 236 in association with the test document name.

[0096] (2A3) Calculation of rule classification result accuracy. Among the records to be evaluated, records with "rule classification result vTre = 0" are defined as "rule classification result correct records." A rule classification result correct record is a test document record in which the rule classification result vTre is a correct value (correct answer). In other words, a rule classification result correct record is a test document record in which the class can be accurately calculated using the rule-based engine RE.

[0097] The ratio of the number of rule classification result correct records to the number of records to be evaluated is calculated as the rule classification result accuracy vrea (rule classification result accuracy vrea=(number of rule classification result correct records) / (number of records to be evaluated)) and saved.

[0098] (2B) Evaluation of the machine classification model MM1. The model evaluation unit 123, which will be described in detail below, performs the following processes (2B1) and (2B2) for each test document name 802 in the evaluation target record. Then, the model update unit 124 performs the process (2B3). (2B1) Using the machine classification model MM1, a machine classification result vTmm is calculated, which is the classification result of the test document associated with the test document name 802. (2B2) A machine classification correctness judgment value vTmmj is calculated, which indicates whether the machine classification result vTmm is a correct value (correct answer). (2B3) Using the machine classification correctness judgment value vTmmj, the proportion of the machine classification result vTmm that is a correct value (the accuracy of the machine classification result vTmm) is calculated as the machine classification result accuracy vmma.

[0099] (2B1) Calculation of the machine classification result vTmm using the machine classification model MM1. The model evaluation unit 123 acquires a test document associated with the test document name of the evaluation target record from the document data store 232. The model evaluation unit 123 classifies the acquired test document using the machine classification model MM1, and calculates the machine classification result vTmm, which is the classification result. The model evaluation unit 123 stores the calculated machine classification result vTmm in the test data information table 236 in association with the test document name.

[0100] (2B2) Calculation of the machine classification accuracy value vTmmj. The model evaluation unit 123 calculates a machine classification correctness judgment value vTmmj indicating whether the machine classification result vTmm is a correct value (correct answer). When the machine classification result vTmm is a correct value (correct answer), the machine classification result vTmm has the same value (class) as the correct answer label of the record to be evaluated. Therefore, when the value of the machine classification result vTmm is the same as the correct answer label, the model evaluation unit 123 sets the value of the machine classification correctness judgment value vTmmj to a value "0" (machine classification correctness judgment value vTmmj = 0) indicating that the machine classification result vTmm is correct. On the other hand, when the value of the machine classification result vTmm is different from the correct answer label, the model evaluation unit 123 sets the value of the machine classification correctness judgment value vTmmj to a value "X" (machine classification correctness judgment value vTmmj = X) indicating that the machine classification result vTmm is incorrect. Then, the model evaluation unit 123 stores the calculated machine classification correctness judgment value vTmmj (0 or X) in the test data information table 236 in association with the test document name.

[0101] (2B3) Calculation of accuracy of machine classification results. Among the records to be evaluated, records for which "machine classification result vTmm = 0" are defined as "machine classification result correct records." A machine classification result correct record is a test document record for which the machine classification result vTmm is a correct value (correct answer). In other words, a machine classification result correct record is a test document record for which the class can be accurately calculated using the machine classification model MM.

[0102] The ratio of the number of correct machine-classified records to the number of records to be evaluated is calculated as the machine classification result accuracy VMMA (machine classification result accuracy VMMA = (number of correct machine-classified records) / (number of records to be evaluated)) and saved.

[0103] (2C) Evaluation of classification accuracy prediction model AP1. The model evaluation unit 123 performs the following processes (2C1) and (2C2) for each test document name 802 of the evaluation target record, as will be described in detail below. Then, the model update unit 124 performs the process (2C3). (2C1) Using the classification accuracy prediction model AP1, calculate the rule classification accuracy predicted value vTap, which is the classification result of the test document associated with the test document name 802. (2C2) Calculate the rule classification accuracy predicted value correctness judgment value vTapj, which indicates whether the rule classification accuracy predicted value vTap is a correct value (rule classification correctness judgment value vTrej, which indicates whether the rule classification result vTre is a correct value or not). (2C3) Using the rule classification accuracy predicted value correctness judgment value vTapj, calculate the rate at which the rule classification accuracy predicted value vTap is a correct value (the accuracy of the rule classification accuracy predicted value vTap) as the rule classification accuracy predicted value accuracy vapa.

[0104] (2C1) Calculation of rule classification accuracy prediction value vTap using classification accuracy prediction model AP1. The model evaluation unit 123 acquires a test document associated with the test document name of the record to be evaluated from the document data store 232. The model evaluation unit 123 classifies the acquired test document using the classification accuracy prediction model AP1, and calculates a rule classification accuracy prediction value vTap, which is the classification result. The model evaluation unit 123 stores the calculated rule classification accuracy prediction value vTap in the test data information table 236 in association with the test document name.

[0105] (2C2) Calculation of the rule classification accuracy prediction value correct / incorrect judgment value vTapj. The model evaluation unit 123 calculates a rule classification accuracy predicted value correctness judgment value vTapj indicating whether the rule classification accuracy predicted value vTap is a correct value (rule classification correctness judgment value vTrej) or not. The rule classification accuracy predicted value vTap is a correct value (correct answer) when the rule classification accuracy predicted value vTap is the same value (O or X) as the rule classification correctness judgment value vTrej of the record to be evaluated. Therefore, when the value of the rule classification accuracy predicted value vTap is the same as the rule classification correctness judgment value vTrej, the model evaluation unit 123 sets the value of the rule classification accuracy predicted value correctness judgment value vTapj to a value "0" (rule classification accuracy predicted value correctness judgment value vTapj=0) indicating that the rule classification accuracy predicted value vTap is correct. On the other hand, when the value of the rule classification accuracy predicted value vTap differs from the rule classification correctness judgment value vTrej, the model evaluation unit 123 sets the value of the rule classification accuracy predicted value correctness judgment value vTapj to a value "X" indicating that the rule classification accuracy predicted value vTap is incorrect (rule classification accuracy predicted value correctness judgment value vTapj=X). Then, the model evaluation unit 123 stores the calculated value of the rule classification accuracy predicted value correctness judgment value vTapj (0 or X) in the test data information table 236 in association with the test document name.

[0106] (2C3) Calculation of rule classification accuracy prediction value probability vapa. Among the records to be evaluated, records in which "rule classification accuracy predicted value correctness judgment value vTapj=0" are defined as "rule classification accuracy predicted value correct records." A rule classification accuracy predicted value correct record is a test document record in which the rule classification accuracy predicted value vTap is a correct value (rule classification correctness judgment value vTrej). In other words, a rule classification accuracy predicted value correct record is a test document record in which the rule classification accuracy predicted value vTap can be accurately calculated using the classification accuracy prediction model AP.

[0107] The ratio of the number of records that are correct in the rule classification accuracy prediction value to the number of records to be evaluated is calculated as the rule classification accuracy prediction value accuracy vapa (rule classification accuracy prediction value accuracy vapa = (number of records that are correct in the rule classification accuracy prediction value) / (number of records to be evaluated)) and saved.

[0108] (3) Classifying documents using the model The document classification support device 100 transmits the rule-based engine RE, the machine classification model MM, and the classification accuracy prediction model AP stored in the model store 233 to the document classification terminal 200. The document classification terminal 200 stores the received rule-based engine RE, the machine classification model MM, and the classification accuracy prediction model AP in the model store 233 of the secondary storage device 23 (storage unit 11).

[0109] The document classification unit 125 can use the rule-based engine RE to calculate the rule classification result vre from a plurality of item values. In addition, the document classification unit 125 can use the rule-based engine RE to calculate rule-use item values, which are item values ​​used for classification, in the process of calculating the rule classification result vre from a plurality of item values ​​of the document. In addition, the document classification unit 125 can use the machine classification model MM to calculate the machine classification result vmm from a plurality of item values ​​of the document. In addition, the document classification unit 125 can use the machine classification model MM to calculate machine-use item values, which are item values ​​used for classification, in the process of calculating the machine classification result from a plurality of item values. In addition, the document classification unit 125 can use the classification accuracy prediction model AP to calculate the rule classification accuracy prediction value vap from a plurality of item values. In addition, the document classification unit 125 can use the classification accuracy prediction model AP to calculate the rule classification accuracy prediction use item value, which is the item value used for classification, in the process of calculating the rule classification accuracy prediction value vap from a plurality of item values ​​of the document.

[0110] When a document classifier inputs the name of a document to be classified into the document classification unit 125 of the document classification terminal 200, the document information acquisition unit 121 acquires multiple item values ​​of the document to be classified from the document data store 232 of the document classification support device 100.

[0111] Then, the document classification unit 125 classifies the document based on the multiple item values ​​of the acquired document using each of the rule-based engine RE, the machine classification model MM, and the classification accuracy prediction model AP, and calculates the classification result (rule classification result vre, machine classification result vmm, and rule classification accuracy prediction value vap), the rule use item value, the machine use item value, and the rule classification accuracy prediction use item value. The document classification unit 125 outputs the calculated classification result (rule classification result vre, machine classification result vmm, and rule classification accuracy prediction value vap), the rule use item value, the machine use item value, the rule classification accuracy prediction use item value, and classification result display screen information including information on the configuration of the classification result display screen to the display unit 13 (a monitor or touch panel that is a part of the output device 25). The display unit 13 displays the classification result display screen.

[0112] Fig. 10 is a diagram showing an example of a classification result display screen. Classification result display screen 1000 shown in Fig. 10 includes a document name display field 1010, a classification result display table 1020, a document data display table 1030, a class determination result input field 1040, a class determination result registration button 1050, and a classification judgment criteria display button 1060.

[0113] The document name display field 1010 is a field that displays the name of the document to be classified. The classification result display table 20 is a table that displays the document classification result using the rule-based engine RE and the rule classification result accuracy vrea, the document classification result using the classification accuracy prediction model AP and the rule classification accuracy predicted value accuracy vapa, and the document classification result using the machine classification model MM and the machine classification result accuracy vmma. The classification result display table 1020 includes a classification method column 1021 with the model name or the like as a value, a classification result 1022, and classification accuracy (rule classification result accuracy vrea, rule classification accuracy predicted value accuracy vapa, machine classification result accuracy vmma) 1023.

[0114] Document data display table 1030 is a table that displays a plurality of item values ​​of documents to be classified. Document data display table 1030 has a column 1031 of items and a column 1032 of item values.

[0115] The class determination result input field 1040 is a field where the document classifier inputs a class as a document classification result. When the class determination result registration button 1050 is pressed, the document classification terminal 200 transmits classification result information including the document type, document name, document item, all item values, and the class information input in the class determination result input field 1040 to the document classification support device 100. When the document classification support device 100 receives the document classification result information, it creates and stores in the document data store 232 a record including the document type, document name, document item, and item value information included in the document classification result information. In addition, the document classification support device 100 creates and stores in the document information table 234 a record including the type of document, document name, and class included in the document classification result information.

[0116] When the classification criteria display button 1060 is pressed, a classification basis display screen (see Figure 11) is displayed that displays the item values ​​(rule-used item values, machine-used item values, rule classification accuracy prediction item values) used by the models (rule-based engine RE, classification accuracy prediction model AP, and machine classification model MM) as the basis for classification when classifying documents.

[0117] A document classifier can determine the class of a document to be classified by referring to classification result display table 1020 and document data display table 1030. Thus, document classification system 1 makes it easy to classify a document, which includes information on item values ​​for each of a plurality of items, into a class based on the plurality of item values.

[0118] Fig. 11 is a diagram showing an example of a classification basis display screen. The classification basis display screen 1100 shown in Fig. 11 includes a rule-based engine RE classification basis display table 1110, a machine classification model classification basis display table 1120, a classification accuracy prediction model AP classification basis display table 1130, a classification result display button 1140, a document name display field 1010 similar to the classification result display screen 1000 of Fig. 10, a classification result display table 1020, a document data display table 1030, a class determination result input field 1040, a class determination result registration button 1050, and a classification judgment criterion display button 1060.

[0119] The rule-based engine RE classification basis display table 1110 is a table that displays rule use item values. The rule-based engine RE classification basis display table 1110 includes items 1111 and rule use item values ​​1112.

[0120] The machine classification model classification basis display table 1120 is a table that displays machine usage item values. The machine classification model classification basis display table 1120 includes items 1121 and machine usage item values ​​1122.

[0121] The classification accuracy prediction model AP classification basis display table 1130 is a table that displays the rule classification accuracy prediction use item values. The classification accuracy prediction model AP classification basis display table 1130 includes an item 1131 and a rule classification accuracy prediction use item value 1132.

[0122] When the sorting result display button 1140 is pressed, the sorting result display screen shown in FIG. 10 is displayed.

[0123] The document name display field 1010, classification result display table 1020, document data display table 1030, class determination result input field 1040, class determination result registration button 1050, and classification criteria display button 1060 are the same as those in the classification result display screen 1000 of Figure 10, so their explanation will be omitted.

[0124] A document classifier can determine the class of a document to be classified by referring not only to the classification result display table 1020 and the document data display table 1030 but also to the rule-based engine RE classification basis display table 1110, the machine classification model classification basis display table 1120, and the classification accuracy prediction model AP classification basis display table 1130. Thus, the document classification system 1 makes it easy to classify a document, which includes information on item values ​​for each of a plurality of items, into a class based on the plurality of item values.

[0125] The classification accuracy prediction model AP can be used to determine which of two calculation results of values ​​indicating the document class, the rule classification result vre calculated using the rule-based engine RE and the machine classification result vmm calculated using the machine classification model MM, is the correct class. The document classification terminal 200 calculates and displays two values, the rule classification result vre calculated using the rule-based engine RE and the machine classification result vmm calculated using the machine classification model MM, as the calculation results of the document class. Even if a document classifier refers to the rule classification result vre and the machine classification result vmm, it is not easy to determine which value, the rule classification result vre or the machine classification result vmm, is correct.

[0126] Therefore, a classification accuracy prediction model AP is created that can be used to predict whether the rule classification result vre calculated by the rule-based engine RE is correct or not. Then, the document classification terminal 200 uses the classification accuracy prediction model AP to calculate a rule classification accuracy prediction value vap that indicates whether a document can be correctly classified using the rule-based engine RE (whether the rule classification result vre is correct or not), and displays the rule classification accuracy prediction value vap.

[0127] If the rule classification accuracy prediction value vap is greater than a predetermined value, the document classifier can determine that the value (class) of the rule classification result vre is sufficiently accurate and regard the value of the rule classification result vre as the class of the document. On the other hand, if the rule classification accuracy prediction value vap is smaller than a predetermined value, the document classifier can determine that the value of the rule classification accuracy prediction value vap is not accurate and regard the value of the machine classification result vmm as the class of the document. In this way, the classification accuracy prediction model AP makes it easy to determine the correct class of a document from two values ​​indicating the class of the document: the rule classification result vre calculated using the rule-based engine RE and the machine classification result vmm calculated using the machine classification model MM.

[0128] (4) Updating the model and evaluating the updated model As described above, when a document classifier inputs the document classification results and the class determination result registration button into the document classification terminal 200, the document data (including information associating the document name, item name, and item value) is stored in the document data store 232 of the document classification support device 100, and a record that stores information on the document classification results (including the document name and class) is stored in the document information table 234 of the document classification support device 100. In this way, when the number of documents classified into classes exceeds a predetermined number (e.g., 1000), the model update unit 124 and the model evaluation unit 123 of the document classification support device 100 update the models (the rule-based engine RE, the classification accuracy prediction model AP, and the machine classification model MM) and evaluate the models as follows.

[0129] First, the model update unit 124 of the document classification support device 100 classifies documents classified into classes into training documents and test documents at a predetermined ratio. Then, the document classification support device 100 creates records including the document names of the documents classified into the training documents and the class (the value of the correct label) which is the classification result, and stores them in the training data information table 235. Also, the document classification support device 100 creates records including the document names of the documents classified into the test documents and the class (the value of the correct label) which is the classification result, and stores them in the test data information table 236. Here, the model update unit 124 stores the class (class 503 of the document data store 232) which is the classification result in the training data information table 235 and the correct labels 703, 803 of the test data information table 236.

[0130] Then, the model update unit 124 performs (4A), updating the machine classification model MM and evaluating the machine classification model MM, (4B), updating the rule-based engine RE and evaluating the rule-based engine RE, and (4C), updating the classification accuracy prediction model AP and evaluating the classification accuracy prediction model AP.

[0131] (4A) Updating the machine classification model MM and evaluating the machine classification model MM. (4A1) Update of the machine classification model MM. Similar to the generation of the machine classification model MM1 (1B) described above, the model update unit 124 trains the machine classification model MM (referred to as the "pre-training machine classification model MMa") using training data (documents with document names stored in records in which a value for the training use date 704 is not stored in the training data information table 235, and correct labels), and sets the trained machine classification model MM as an updated machine classification model MMb. Then, the model update unit 124 stores the updated machine classification model MMb in the model store 233. Here, by training the machine classification model MM1a, a machine classification model MM1 is generated, which is a machine classification model MM1 that has been trained using the training data from the machine classification model MM1a.

[0132] (4A2) Evaluation of the updated machine classification model MM. The model update unit 124 performs processing on the updated machine classification model MMb similar to the above-mentioned (2B) evaluation of the machine classification model MM1, calculates and stores the machine classification result accuracy vmma for the updated machine classification model MMb. Through the processing described above, the updated machine classification model MMb is generated and stored in the model store 233 to update the machine classification model MM, and the value of the machine classification result Tvmm for the test data and the machine classification correctness judgment value vTmmj are stored in the test data information table 236.

[0133] (4B) Updating the rule-based engine RE and evaluating the rule-based engine RE. In updating the rule-based engine RE, additional classification rules are incorporated into the rule-based engine RE. The additional classification rules are rules that were not incorporated into the rule-based engine RE before the update. The additional classification rules are also rules that are incorporated into the machine classification model MM after the update.

[0134] A document that can be classified using the additional classification rules is a document that cannot be correctly classified into a class by the rule-based engine RE before the update, and can be correctly classified into a class by the machine classification model MM after the update.

[0135] In updating the rule-based engine RE, the model update unit 124 executes the following processes (4B1) onward.

[0136] (4B1) The model update unit 124 detects (A) a document that cannot be correctly classified into a class by the rule-based engine RE before the update, and (B) a document that can be correctly classified into a class by the updated machine classification model MMb (referred to as a document to be corrected). That is, (4B1A) the model update unit 124 classifies the test document using the rule-based engine RE before the update, and stores the rule classification result vTre, which is the classification result, in the rule classification result vre805 of the test data information table 236 in association with the test document name. Also, in the evaluation of the updated machine classification model MM in (4A2) described above, the model update unit 124 calculates the machine classification result vTmm, which is the classification result of the test document using the updated machine classification model MMb, and stores it in the machine classification result vTmm809 of the test data information table 236.

[0137] The model update unit 124 calculates the names of test documents that cannot be classified correctly into classes by the rule-based engine RE before the update and can be classified correctly into classes by the machine classification model MM after the update, among the test documents. That is, the model update unit 124 searches the test data information table 236 for records in which the value of the test use date 804 is not stored, and obtains, from among the searched records, records in which the value of the rule classification result vTre 805 of the record is different from the correct label 803 (the rule-based engine RE before the update cannot classify the record correctly into a class) and the value of the machine classification result vTmm of the record is the same as the correct label 803 (the record can be classified correctly into a class by the machine classification model MM after the update) (referred to as "records to be updated"). Then, the model update unit 124 generates a list of document names to be updated that lists the test document names 802 of the records to be updated.

[0138] (4B2) Creating additional classification rules and adding the additional classification rules to the rule-based engine RE. The model update unit 124 performs the following processes (4B21) to (4B23) for each test document name stored in the document name list to be updated, and generates an updated rule-based engine RE. Then, the model update unit 124 stores the updated rule-based engine RE in the model store 233.

[0139] (4B21) Information on the classification method for the document to be corrected into classes (referred to as "classification method information") is extracted from the updated machine classification model MM and displayed. Here, the model update unit 124 can use the rule-based engine RE to calculate rule classification conditions for classifying item values ​​in the process of calculating a rule classification result from multiple item values ​​of the document. Also, using the machine classification model, it can calculate machine classification conditions for classifying item values ​​in the process of calculating a machine classification result from multiple item values.

[0140] The administrator inputs multiple item values ​​of the correction document into the input device 24 of the document classification support device 100, such that the rule classification result calculated using the rule-based engine RE is an incorrect value, and the machine classification result calculated using the machine classification model is a correct value.

[0141] When multiple item values ​​of a correction document are input, for which the rule classification result calculated using the rule-based engine RE is an incorrect value and the machine classification result calculated using the machine classification model is a correct value, the model update unit 124 calculates rule classification conditions and machine classification conditions for the multiple item values ​​of the correction document, calculates classification difference information, which is information on the difference between the calculated rule classification conditions and the machine classification conditions, and outputs the rule classification conditions, machine classification conditions, and classification difference information to the display unit 13.

[0142] (4B22) The administrator creates appropriate additional classification rules from the “classification method information” and inputs them to the model update unit 124 of the document classification support apparatus 100.

[0143] (4B23) A correction classification command created based on the classification difference information is added to the rule-based engine RE to correct the rule-based engine. The model update unit 124 generates corrected classification commands including conditional branches such as If-Then from the input additional classification rules, and incorporates the generated corrected classification commands into the rule-based engine RE before the update.

[0144] 12A and 12B are explanatory diagrams for explaining the update of the rule-based engine RE. FIG. 12A is a tree diagram showing, in a tree format, the document class classification rules incorporated in the rule-based engine RE before the update. The tree diagram in FIG. 12A has a plurality of judgment conditions 1210, a plurality of classification results 1220, and a speech bubble 1230. The judgment conditions 1210 have an item name (e.g., item 2) and a judgment condition (e.g., "==False" representing a condition branch as to whether the item value is "False" or not, and ">100" representing a condition branch as to whether the item value is greater than 100 or not) of the judgment condition in the class classification rule. The classification result 1220 shows the class that is the document classification result. The speech bubble 1230 includes an additional classification rule (a rule that is not incorporated in the rule-based engine RE before the update and is incorporated in the machine classification model MMb after the update) 1231 ("=="O"" in the example of FIG. 12A), a change command button 1232, and a no change command button 1233. The additional classification rule 1231 indicates a judgment condition of a part of the additional classification rule that differs from a rule incorporated in the rule-based engine RE. When the change command button 1232 is pressed, the model update unit 124 incorporates the judgment condition indicated in the additional classification rule 1231 into the rule-based engine RE. On the other hand, when the no change command button 1233 is pressed, the model update unit 124 does not incorporate the judgment condition indicated in the additional classification rule 1231 into the rule-based engine RE.

[0145] The administrator can consider whether to incorporate the judgment conditions shown in the additional classification rules 1231 into the rule-based engine RE by referring to the judgment conditions shown in the additional classification rules 1231. The administrator can input an instruction as to whether to incorporate the judgment conditions shown in the additional classification rules 1231 into the rule-based engine RE by pressing a change command button 1232 or a no change command button 1233.

[0146] The tree diagram in FIG. 12B is an example of a tree diagram indicating that the administrator has pressed the change command button 1232, and thus the judgment conditions indicated in the additional classification rule 1231 have been incorporated into the rule-based engine RE.

[0147] (4B3) Evaluation of the updated rule-based engine RE. The model update unit 124 performs processing similar to the processing of evaluation of the rule-based engine RE1 in (2A) described above, calculates the rule classification result accuracy vrea of ​​the updated rule-based engine RE, and stores the calculated rule classification result accuracy vrea.

[0148] (4C) Update of the classification accuracy prediction model AP and evaluation of the classification accuracy prediction model AP. (4C1) Update of classification accuracy prediction model AP. The model update unit 124 calculates the rule classification result vLre and the rule classification correctness judgment value vLrej using the updated rule-based engine RE, the training document, and the training data information table 235, similar to the above-mentioned (1C) generation of the classification accuracy prediction model AP1 (steps S120 to S160 of the flowchart of the classification accuracy prediction model generation process in FIG. 9), and stores them in the training data information table 235. Then, similar to step S170 of the flowchart of the classification accuracy prediction model generation process in FIG. 9, the model update unit 124 trains the classification accuracy prediction model AP before the update using the rule classification correctness judgment value vLrej and the training document stored in the training data information table 235 to generate the classification accuracy prediction model AP after the update. Here, by training the classification accuracy prediction model AP before the update, the classification accuracy prediction model AP before the update is trained using the training data (the rule classification correctness judgment value vLrej and the training document), and the updated classification accuracy prediction model AP is generated. (4C2) Evaluation of the updated classification accuracy prediction model AP. The model update unit 124 performs processing similar to the above-mentioned (2C) evaluation of the classification accuracy prediction model AP1 on the updated classification accuracy prediction model AP, and calculates and stores the rule classification accuracy prediction value accuracy vapa for the updated classification accuracy prediction model AP.

[0149] <Effects of the Invention> As described above, the document classification system 1 calculates and outputs not only the rule classification result vre and the machine classification result vmm calculated by the document classification unit 125, but also the predicted rule classification accuracy value vap. This allows the document classification system 1 to display not only the rule classification result vre and the machine classification result vmm, but also the predicted rule classification accuracy value vap.

[0150] If a document classifier classifies a document into a class by referring to the rule classification result vre and the machine classification result vmm, which are values ​​that indicate the class, without referring to the rule classification accuracy predicted value vap, it is difficult to determine which value, the rule classification result vre or the machine classification result vmm, should be given priority for reference.

[0151] The document classification system 1 can display not only the rule classification result vre and the machine classification result vmm but also the rule classification accuracy prediction value vap in the document classification unit 125. Then, based on the displayed rule classification accuracy prediction value vap, the document classifier can classify documents into classes by considering which value is more useful, the rule classification result vre calculated using the rule-based engine RE or the machine classification result vmm calculated using the machine classification model MM.

[0152] For example, if the rule classification accuracy prediction value vap calculated using the classification accuracy prediction model AP indicates that the document can be correctly classified using the rule-based engine RE (rule classification accuracy prediction value vap = O), the document classifier can determine that the validity of the class indicated by the rule classification result vre is high and classify the document into the class indicated by the rule classification result vre. Also, if the rule classification accuracy prediction value vap calculated using the classification accuracy prediction model AP indicates that the document cannot be correctly classified using the rule-based engine RE (rule classification accuracy prediction value vap = X), the document classifier can determine that the validity of the class indicated by the machine classification result vmm is high and classify the document into the class indicated by the machine classification result vmm. Thus, the document classification system 1 makes it easy to classify a document including information on item values ​​for each of a plurality of items into classes based on the plurality of item values.

[0153] The rule-based engine RE is created based on written rules (examination criteria, etc.) for classifying documents. When the document classification system 1 uses the rule-based engine RE to classify documents, it can classify documents using written rules, which are explicit knowledge, that are incorporated into the rule-based engine RE. On the other hand, the machine classification model MM is a machine learning model. When the document classification system 1 uses the machine classification model MM to classify documents, it can classify documents using the explicit and tacit knowledge rules that the machine classification model MM has learned.

[0154] The document classifier can classify documents based on a classification result with high reliability among not only the rule classification result vre (classification result using the rule-based engine RE) based on explicit rules, which is explicit knowledge, but also the machine classification result vmm based on the classification method learned by the machine classification model MM, which is explicit knowledge and tacit knowledge. This makes it easier for the document classifier to classify documents that are not easy to classify based on explicit rules into classes. Therefore, the document classification system 1 makes it easier to classify documents that include multiple item values ​​for multiple items and that are not easy to classify based on explicit rules into classes based on the multiple item values.

[0155] Furthermore, the document classification assistance device 100 can reduce the energy required for classifying documents and the amount of carbon dioxide emissions generated, thereby suppressing global warming.

[0156] The training data also includes rule-classifiable group data and rule-incompatible group data. The rule-classifiable group data is data of pairs of multiple item values ​​of a document that can be correctly classified using the rule-based engine RE and correct labels (correct classes). The rule-incompatible group data is data of pairs of multiple item values ​​of a document that cannot be correctly classified even using the rule-based engine RE and correct labels (correct classes).

[0157] The machine classification model MM is a trained model trained using training data. If a document to be classified can be correctly classified into a class using the rule-based engine RE and the value of the rule classification accuracy prediction value vap presented to the document classifier indicates that the document cannot be correctly classified using the rule-based engine RE, the machine classification result vmm can be used to calculate an appropriate class for the document, so that the document classifier can refer to the machine classification result vmm to more appropriately classify the document into a class. Thus, the document classification system 1 facilitates classifying a document, which includes information on item values ​​for each of a plurality of items of the document, into a more appropriate class based on the plurality of item values.

[0158] Furthermore, when multiple item values ​​of the document to be corrected are input, in which the rule classification result vre calculated using the rule-based engine RE is an incorrect value and the machine classification result vmm calculated using the machine classification model MM is a correct value, the model update unit 124 calculates rule classification conditions and machine classification conditions for the multiple item values ​​of the document to be corrected. The model update unit 124 calculates classification difference information, which is information on the difference between the calculated rule classification conditions and the machine classification conditions. Furthermore, the model update unit 124 outputs the rule classification conditions, the machine classification conditions, and the classification difference information to the display unit 13. Then, the display unit 13 displays the rule classification conditions, the machine classification conditions, and the difference information.

[0159] The rule classification conditions are conditions for classifying item values ​​in the process of calculating a rule classification result vre from multiple item values ​​of a document using a rule-based engine RE, and the machine classification conditions are conditions for classifying item values ​​in the process of calculating a machine classification result vmm from multiple item values ​​of a document using a machine classification model MM.

[0160] The administrator creates additional classification rules to be added to the rule-based engine RE based on the rule classification conditions, machine classification conditions, and classification difference information for multiple item values ​​of the document to be corrected displayed on the display unit 13. Then, the model update unit 124 generates a correction classification command from the additional rules. The model update unit 124 adds the correction classification command to the rule-based engine RE to correct the rule-based engine RE. The corrected rule-based engine RE is used to classify documents.

[0161] Here, the rule-based engine RE is created based on written rules (examination criteria, etc.) for classifying documents. The rule-based engine RE incorporates written rules, which are explicit knowledge. Meanwhile, the machine classification model MM incorporates explicit and implicit rules learned by the machine classification model MM. The document classification system 1 incorporates into the rule-based engine RE rules for classifying documents (additional classification rules) that are incorporated into the machine classification model MM but are also incorporated into the rule-based engine RE. This makes it possible to explicitly incorporate the implicit rules incorporated into the machine classification model MM into the rule-based engine RE, thereby improving the accuracy of document classification by the rule-based engine RE.

[0162] In this manner, by correcting the rule-based engine RE, the document classification system 1 can easily correct the rule-based engine RE so that a document that could not be correctly classified using the rule-based engine RE can be classified. As a result, the document classification system 1 can easily classify a document that includes information on item values ​​for each of a plurality of items into a more appropriate class based on the plurality of item values.

[0163] In addition, the machine classification model MM is a decision tree model. Therefore, the model update unit 124 can easily calculate machine classification conditions for classifying item values ​​in the process of calculating the machine classification result vmm from multiple item values ​​of a document using the machine classification model MM. Furthermore, it becomes easy to correct the rule-based engine RE.

[0164] Furthermore, the document classification unit 125 calculates rule use item values, which are item values ​​used for classification, in the process of calculating the rule classification result vre, and machine use item values, which are item values ​​used for classification, in the process of calculating the machine classification result vmm, and outputs these to the display unit 13. This allows the display unit 13 to display the rule classification result vre, the rule use item values, the machine classification result vmm, the machine use item values, and the rule classification accuracy prediction value vap. Then, the document classifier can classify documents into classes by taking into account the rule use item values ​​and machine use item values ​​that are the basis for the calculation when calculating the rule classification result vre and the machine classification result vmm. Thus, the document classification system 1 makes it easy to classify documents into classes.

[0165] Furthermore, when multiple item values ​​of a document are input, the document classification system 1 causes the document classification unit 125 to calculate the rule classification result vre, the machine classification result vmm, and the rule classification accuracy predicted value vap. This allows the display unit 13 to display the rule classification result vre, the rule classification result accuracy vrea, the machine classification result vmm, the rule classification result accuracy vrea, and the machine classification result accuracy vmma. The document classifier can then classify the document by considering the accuracy of the rule classification result vre based on the rule classification result accuracy vrea, and further considering the accuracy of the machine classification result vmm based on the machine classification result accuracy vmma. As a result, it becomes even easier to accurately classify documents into classes. Thus, the document classification system 1 makes it easier to classify documents into classes.

[0166] <Modification> 2, the document classification terminal 200 of the embodiment described above has a storage unit 11 that stores the model store 233, a document information acquisition unit 121, a document classification unit 125, a display unit 13, and a scanner 241. In the document classification system 1 of the modified example, the document classification support device 100 is a server. Moreover, instead of using the model store 233 and the document classification unit 125 of the document classification terminal 200, the document classification terminal 200 of the modified example accesses the document classification support device 100 and uses the model store 233 and the document classification unit 125 of the document classification support device 100 to classify documents.

[0167] In addition, each of the above configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that realizes the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-mentioned embodiments, and the program code itself and the storage medium storing it constitute the present invention. Examples of storage media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs (Solid State Drives), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0168] Furthermore, the program code for realizing the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, Python, Java (registered trademark), etc.

[0169] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed over a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read out and execute the program code stored in the storage means or storage medium.

[0170] In the above-mentioned embodiment, the control lines and information lines are shown as those considered necessary for the explanation, and not all the control lines and information lines are shown in the product. All the components may be connected to each other. [Explanation of symbols]

[0171] 1: Document classification system 100: Document classification support device 11: Storage part 12: Processing section 121: Document information acquisition unit 122: Model Creation Department 123: Model evaluation section 124: Model Update Section 125: Document Classification Department 126: Document Preparation Department 13:Display section 121a: Document information acquisition program 122a: Model creation program 123a: Model Evaluation Program 124a: Model Update Program 125a: Document classification program 126a: Document creation programs 21: Processor 22: Main memory 23: Secondary storage device 231: Standard rule database 232: Document data store 233: Model Store 234: Document information table 235: Training data information table 236: Test data information table 24: Input device 25: Output device 26: Network I / F 27: Bus 200: Document sorting terminal 500: General terminal

Claims

1. A document classification system that classifies a document including a plurality of item values ​​for a plurality of items into a class based on the plurality of item values ​​of the document, The document classification system includes a memory unit and a processing unit. The storage unit is a rule-based engine that is a model representing a relationship between the multiple item values ​​of the document and the class, the model being created based on a classification rule for classifying the document into the class based on the multiple item values ​​of the document; a classification accuracy prediction model that is a machine learning model that represents a relationship between the multiple item values ​​of the document and a rule classification accuracy prediction value that represents whether the document can be correctly classified using the rule-based engine; a machine classification model that is a machine learning model representing a relationship between the plurality of item values ​​of the document and the class; Store the The processing unit includes: When the plurality of item values ​​of the document are input, Calculating a rule classification result, which is a classification result of the document, from the multiple item values ​​of the input document using the rule-based engine; calculating a machine classification result, which is a classification result of the document, from the multiple item values ​​of the input document using the machine classification model; Calculating the rule classification accuracy prediction value from the multiple item values ​​of the input document using the classification accuracy prediction model; outputting the calculated rule classification result, the machine classification result, and the rule classification accuracy prediction value; Document classification system.

2. 2. The document classification system according to claim 1, A machine classification model is a learned model that is trained using training data. The training data includes rule classifiable pair data of pairs of training documents and correct labels that can be correctly classified into classes using the rule-based engine, and rule unclassifiable pair data of pairs of training documents and correct labels that cannot be correctly classified into classes using the rule-based engine. Document classification system.

3. 2. The document classification system according to claim 1, The processing unit further includes a model update unit, The model update unit is Using the rule-based engine, a rule classification condition for classifying the item values ​​in the process of calculating the rule classification result from the multiple item values ​​of the document can be calculated; Using the machine classification model, a machine classification condition for classifying the item values ​​can be calculated in the process of calculating the machine classification result from the plurality of item values; When the multiple item values ​​of the correction document are input, the rule classification result calculated using the rule-based engine is an incorrect value, and the machine classification result calculated using the machine classification model is a correct value, Calculating the rule classification conditions and the machine classification conditions for the plurality of item values ​​of the correction document; Calculating classification difference information, which is information on the difference between the calculated rule classification condition and the calculated machine classification condition; outputting the rule classification condition, the machine classification condition, and the classification difference information; applying corrected classification instructions generated based on the classification difference information to the rule-based engine to correct the rule-based engine; Document classification system.

4. 4. The document classification system according to claim 3, The machine classification model MM is a decision tree model. Document classification system.

5. 2. The document classification system according to claim 1, The processing unit includes a document classification unit that classifies the document into the class, The document classification unit includes: In the process of calculating the rule classification result from the plurality of item values ​​using the rule-based engine, a rule use item value that is the item value used for classification can be calculated; In the process of calculating the machine classification result from the plurality of item values ​​using the machine classification model, a machine use item value, which is the item value used for classification, can be calculated; When the plurality of item values ​​of the document are input, calculating the rule classification result and the rule use item value from the plurality of item values ​​using the rule-based engine; calculating the machine classification result and a machine usage item value from the plurality of item values ​​using the machine classification model; calculating the rule classification accuracy prediction value from the plurality of item values ​​using the classification accuracy prediction model; outputting the calculated rule classification result, the rule use item value, the machine classification result, the machine use item value, and the rule classification accuracy prediction value; Document classification system.

6. 2. The document classification system according to claim 1, The processing unit includes a model evaluation unit, The model evaluation unit includes: For each of the test documents included in the test data including a plurality of test set data of a pair of a test document and a class of the test document which is a correct answer label, Using the rule-based engine, calculate a rule classification result for the test document from a plurality of item values ​​of the test document; calculating a machine classification result for the test document from a plurality of item values ​​of the test document using the machine classification model; Calculate the rate of rule classification results that are the same as the correct label to the number of all rule classification results calculated as the rule classification result accuracy; Calculate the accuracy of the machine classification result as the ratio of the machine classification results that are the same as the correct label to the number of all the machine classification results. The document classification unit includes: When the plurality of item values ​​of the document are input, Calculating a rule classification result, which is a classification result of the document, from the multiple item values ​​of the input document using the rule-based engine; calculating a machine classification result, which is a classification result of the document, from the multiple item values ​​of the input document using the machine classification model; Calculating the rule classification accuracy prediction value from the multiple item values ​​of the input document using the classification accuracy prediction model; outputting the calculated rule classification result, the machine classification result, the rule classification accuracy prediction value, the rule classification result accuracy, and the machine classification result accuracy; Document classification system.

7. A document classification method in a document classification system comprising a storage device and a processor, the processor classifying a document including a plurality of item values ​​for a plurality of items into a class based on the plurality of item values ​​of the document, comprising: The storage device includes: a rule-based engine that is a model representing a relationship between the multiple item values ​​of the document and the class, the model being created based on a classification rule for classifying the document into the class based on the multiple item values ​​of the document; a classification accuracy prediction model that is a machine learning model that represents a relationship between the multiple item values ​​of the document and a rule classification accuracy prediction value that represents whether the document can be correctly classified using the rule-based engine; a machine classification model that is a machine learning model representing a relationship between the plurality of item values ​​of the document and the class; Store the The processor, When the plurality of item values ​​of the document are input, Calculating a rule classification result, which is a classification result of the document, from the multiple item values ​​of the input document using the rule-based engine; calculating a machine classification result, which is a classification result of the document, from the multiple item values ​​of the input document using the machine classification model; Calculating the rule classification accuracy prediction value from the multiple item values ​​of the input document using the classification accuracy prediction model; outputting the calculated rule classification result, the machine classification result, and the rule classification accuracy prediction value; Document classification method.

8. A document classification program for a document classification system including a storage device and a processor, the processor classifying a document including a plurality of item values ​​for a plurality of items into a class based on the plurality of item values ​​of the document, The storage device includes: a rule-based engine that is a model representing a relationship between the multiple item values ​​of the document and the class, the model being created based on a classification rule for classifying the document into the class based on the multiple item values ​​of the document; a classification accuracy prediction model that is a machine learning model that represents a relationship between the multiple item values ​​of the document and a rule classification accuracy prediction value that represents whether the document can be correctly classified using the rule-based engine; a machine classification model that is a machine learning model representing a relationship between the plurality of item values ​​of the document and the class; Store the The document classification program includes: When the plurality of item values ​​of the document are input, The processor, Using the rule-based engine, a rule classification result is calculated, which is a classification result of the document, from the multiple item values ​​of the input document; calculating a machine classification result, which is a classification result of the document, from the multiple item values ​​of the input document using the machine classification model; calculating the rule classification accuracy prediction value from the multiple item values ​​of the input document using the classification accuracy prediction model; outputting the calculated rule classification result, the machine classification result, and the rule classification accuracy prediction value; Document sorting program.

Images