Document classification system, document classification device, and document classification method

Abstract

To provide a document classification system, a document classification apparatus, and a document classification method for presenting the grounds for classification based on a classification criterion of document classification.SOLUTION: In a document classification system 1, a document classification apparatus (classification model learning server 30, document classification server 40) includes: a reference feature extraction unit 310 which extracts reference feature quantity of classification from classification reference data having a plurality of feature quantities; a classification model learning unit 320 which generates a document classification model using the extracted reference feature quantity; a document classification unit 420 which classifies a group of documents designated using the generated document classification model; and a grounds generation unit 430 which generates information representing the grounds for classification of the group of documents using the reference feature quantity. A classification client 20 includes: a classification request input unit 210 which receives a request for executing designation of a group of document and classification on the group of documents from a user; and a classification result output unit (classification result display unit 220) which outputs a result of classifying the group of documents classified by the document classification apparatus and information on the grounds for classification.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a document classification system, a document classification apparatus, and a document classification method, and is suitable for application to a document classification system, a document classification apparatus, and a document classification method for classifying a document set.

Background Art

[0002] With the spread of computers and the Internet, a large amount of document information can be obtained. In order to efficiently process a large number of documents, it is effective to utilize a document classification technique for classifying a document set into predetermined categories.

[0003] For example, when it is desired to exclude spam emails from received emails, two categories of spam and non-spam are defined, a classification model of a document classifier is learned using teacher data, and the received emails are classified into spam and non-spam using the learned classification model. Classification into two categories such as spam and non-spam is called binary classification.

[0004] Another example is that when applying for a qualification with grades, categories corresponding to the number of grades of the qualification (for example, five categories from grade 1 to grade 5) are defined, a classification model of a document classifier is learned using teacher data, and the qualification application documents are classified into grades using the learned classification model. Classification into such a plurality of categories is called multi-class classification. Classifying newspaper articles into categories such as "sports", "politics", "international", "society", etc. is also one of multi-class classifications.

[0005] In some cases, the classification result output by the classifier may be accepted without any explanation, but when the influence of the classification result is large, an explanation may be required as to why such a classification result was obtained.

[0006] For example, Non-Patent Document 1 discloses a method for presenting an explanation of the classification result by approximating the classification model with a linear model. Since the linear model is expressed as a weighted sum of the features used for classification, a human can determine which features are effective in the classification result. Also, Patent Document 1 discloses a method for presenting the constituent elements that contributed to the classification of a document using the structural information of the document. Furthermore, Non-Patent Document 2 discloses a method for presenting a part of a document as the basis for classification using the attention mechanism in deep learning. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2021-43849 [Non-patent literature]

[0008] [Non-Patent Document 1] Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin, ““Why Should I TrustYou?” Explaining the Prediction of Any Classifier,” Proceedings of the22nd ACM SIGKDD, 2017 [Non-Patent Document 2] Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, EduardHovy, "HierarchicalAttention Networks for Document Classification", Proceedings of HLT-NAACL, 2016 [Overview of the Initiative] [Problems that the invention aims to solve]

[0009] However, the methods described in the aforementioned prior art documents present classification justifications from the perspective of the machine learning algorithms used by document classifiers, and the justifications presented sometimes deviated from the justifications that humans would consider. This is because document classifiers present classification justifications from the perspective of statistical contributions to classification, while humans expect semantic classification criteria underlying the classification to be presented as justifications. In other words, humans can gain acceptance (or, to put it another way, understand the connection between the classification justification and the classification result) when classification justifications that utilize semantic classification criteria are presented, but the methods described in the prior art documents failed to present such convincing classification justifications.

[0010] The present invention has been made in consideration of the above points, and aims to propose a document classification system, a document classification device, and a document classification method that can present a basis for classification in accordance with the classification criteria for document classification in document classification for classifying a collection of documents. [Means for solving the problem]

[0011] To solve the above problems, the present invention provides a document classification system for classifying a set of documents, comprising: a classification client that receives a request from a user to specify a set of documents and to perform classification on said set of documents; and a document classification device that classifies the set of documents specified by the classification client. A reference feature setting client that receives instructions from the user regarding the addition or deletion of features to the classification criterion data held by the document classification device, The document classification device is equipped with, A criterion generation unit that generates classification criterion data having multiple features from classification training data, and the generated The classification client comprises: a feature extraction unit that extracts feature criteria for classification from classification criterion data; a classification model learning unit that generates a document classification model using the extracted feature criteria; a document classification unit that classifies the document set using the generated document classification model; and a basis generation unit that generates information representing the basis for the classification of the document set by the document classification unit using the feature criteria. The classification client also has a classification result output unit that outputs the classification result of the document set by the document classification unit and the information representing the basis for the classification generated by the basis generation unit. Furthermore, if the addition or deletion of features is specified in the standard feature setting client, the standard feature extraction unit in the document classification device modifies the classification standard data generated by the standard generation unit according to the specification in the standard feature setting client, and extracts the classification standard features using the modified classification standard data. A document classification system characterized by the above is provided.

[0012] Furthermore, in order to solve these problems, the present invention provides a document classification device for classifying a collection of documents, A criterion generation unit that generates classification criterion data having multiple features from classification training data, and the generated The system comprises: a feature extraction unit that extracts feature criteria for classification from classification criterion data; a classification model learning unit that generates a document classification model using the extracted feature criteria; a document classification unit that classifies a set of documents specified by a user using the generated document classification model; and a basis generation unit that generates information representing the basis for the classification of the document set by the document classification unit using the feature criteria. When the user provides instructions regarding the addition or deletion of features to the classification criterion data generated by the criterion generation unit, the criterion feature extraction unit modifies the classification criterion data generated by the criterion generation unit in accordance with the instructions and extracts classification criterion features using the modified classification criterion data. A document classification device characterized by the above is provided.

[0013] Furthermore, in order to solve the above problems, the present invention provides a document classification method using a document classification system for classifying a set of documents, wherein the document classification system includes a classification client that receives a request from a user to specify a set of documents and to perform classification on said set of documents, and a document classification device that classifies the set of documents specified by the classification client. A reference feature setting client that receives instructions from the user regarding the addition or deletion of features to the classification criterion data held by the document classification device, The classification client has a classification request input step in which it receives the designation of the document set and the request to perform the classification, The document classification device includes a criterion generation step in which it generates classification criterion data having multiple features from classification training data, The document classification device, The base generated in the aforementioned base generation step The system comprises: a feature extraction step for extracting feature criteria for classification from classification criterion data; a classification model learning step in which the document classification device generates a document classification model using the feature criteria extracted in the feature extraction step; a document classification step in which the document classification device classifies a set of documents specified in the classification request input step using the document classification model generated in the classification model learning step; a basis generation step in which the document classification device generates information representing the basis for the classification of the set of documents in the document classification step using the feature criteria; and a classification result output step in which the classification client outputs the classification result of the set of documents by the document classification step and the information representing the basis for the classification generated in the basis generation step. If the addition or deletion of features is specified in the criterion feature setting client, the document classification device modifies the classification criterion data generated in the criterion generation step according to the specification, and extracts the classification criterion features using the modified classification criterion data. A document classification method characterized by the above is provided. [Effects of the Invention]

[0014] According to the present invention, in document classification for classifying a document set, it is possible to present a classification basis in accordance with the classification criteria of the document classification.

Brief Description of the Drawings

[0015] [Figure 1] It is a diagram showing a configuration example of a document classification system 1 according to a first embodiment of the present invention. [Figure 2] It is a diagram showing a configuration example of a classification request input screen. [Figure 3] It is a diagram showing a configuration example of a classification result display screen. [Figure 4] It is a sequence diagram showing an example of a processing procedure of document classification processing by the document classification system 1. [Figure 5] It is a diagram showing a configuration example of a document classification system 2 according to a second embodiment of the present invention. [Figure 6] It is a diagram showing a configuration example of a reference feature amount setting screen. [Figure 7] It is a sequence diagram showing an example of a processing procedure of document classification processing by the document classification system 2.

Modes for Carrying Out the Invention

[0016] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0017] Note that the following description and drawings are examples for explaining the present invention, and for the sake of clarity of explanation, appropriate omissions and simplifications are made. Also, not all combinations of features described in the embodiments are essential for the solution means of the invention. The present invention is not limited to the embodiments, and all application examples that conform to the idea of the present invention are included in the technical scope of the present invention. Those skilled in the art can make various additions and changes within the scope of the present invention. The present invention can also be implemented in various other forms. Unless otherwise limited, each component may be plural or singular.

[0018] Furthermore, while the following description may include explanations of processes performed by executing a program, the processor may be the primary entity performing the processing, as a program is executed by at least one processor (e.g., a CPU) and performs defined processes using appropriate memory resources (e.g., memory) and / or interface devices (e.g., communication ports). Similarly, the primary entity performing the processing by executing a program may be a controller, device, system, computer, node, storage system, storage device, server, management computer, client, or host having a processor. The primary entity performing the processing by executing a program (e.g., a processor) may include hardware circuits that perform some or all of the processing. For example, the primary entity performing the processing by executing a program may include hardware circuits that perform encryption and decryption, or compression and decompression. The processor operates as a functional unit that realizes predetermined functions by operating according to the program. Devices and systems including a processor are devices and systems including these functional units.

[0019] A program may be installed from its program source into a device such as a computer. The program source may be, for example, a program distribution server or a computer-readable storage medium. If the program source is a program distribution server, the program distribution server includes a processor (e.g., a CPU) and memory resources, which may further store the distribution program and the program to be distributed. The processor of the program distribution server may then execute the distribution program, thereby distributing the program to other computers. Furthermore, in the following description, two or more programs may be implemented as a single program, or one program may be implemented as two or more programs.

[0020] (1) First Embodiment Figure 1 is a diagram showing an example configuration of a document classification system 1 according to the first embodiment of the present invention. As shown in Figure 1, the document classification system 1 is configured by connecting a classification client 20 used for inputting classification requests by users and displaying classification results, a classification model learning server 30 for learning classification models, and a document classification server 40 for classifying given documents, via a communication network 10.

[0021] In Figure 1, the classification client 20, the classification model learning server 30, and the document classification server 40 are shown as separate devices. However, the document classification system 1 may consist of some or all of these devices on the same computer. For example, the classification model learning server 30 and the document classification server 40 can be called a document classification device if they are configured on the same computer.

[0022] The classification client 20 includes a classification request input unit 210 that receives classification requests from users and transmits those classification requests to the document classification server 40, and a classification result display unit 220 that displays the classification results and classification justifications received from the document classification server 40. In the classification request from the user, the set of documents to be classified (target documents) is specified by the user. In the following description, the classification result display unit 220 displays the classification results and classification justifications on the display of the classification client 20, but this is only one form of output by the classification client 20, and it may also be printed or output as data. Furthermore, the output destination is not limited to the display of the classification client 20, but may be any display device or storage medium.

[0023] The classification model learning server 30 includes a reference feature extraction unit 310 that extracts reference features from document classification training data 330 and classification criterion data 340, and a classification model learning unit 320 that generates a document classification model 350 using the reference features extracted by the reference feature extraction unit 310.

[0024] The document classification training data 330 is training data used for document classification. The classification criterion data 340 is data indicating the classification criteria for document classification, and in the first embodiment, it is assumed to be prepared and set in advance. The document classification model 350 is a model generated by learning by the classification model learning unit 320 as described above, and is a classification model that performs document classification.

[0025] The algorithm used by the criterion feature extraction unit 310 can be any known method that can associate the text contained in the classification criterion data 340 with the text of each document contained in the document classification training data 330. For example, words contained in the classification criterion data 340 can be associated with words contained in the document classification training data 330 using a string matching method. Alternatively, similarity can be calculated using the edit distance between strings to perform similarity-based association. Alternatively, similarity-based association can be performed using a separate thesaurus or the like, with string matching methods or similarity calculations using edit distance. Alternatively, word embedding representation generation methods such as Word2Vec or GloVe can be used to vectorize words, calculate the distance between words using the similarity between vectors, and perform similarity-based association. Alternatively, association may be performed at the phrase level instead of word level. Alternatively, association may be performed at the sentence level. When performing sentence-level association, the Sentence BERT sentence embedding representation generation method can also be used.

[0026] The classification model learning unit 320 can use any publicly known classification learning algorithm (classification learner). Specifically, for example, it can use publicly known techniques such as the decision tree algorithm, the naive Bayes algorithm, and language model-based methods such as BERT.

[0027] The document classification server 40 includes a standard feature extraction unit 410 that extracts standard features from documents sent from the classification client 20 and classification standard data 440, a document classification unit 420 that classifies the documents using the standard features extracted by the standard feature extraction unit 410 and a document classification model 450, and a basis generation unit 430 that generates information (classification basis) that represents the basis for the classification by the document classification unit 420.

[0028] The evidence generation unit 430 can use any known method for generating evidence. For example, if a decision tree algorithm is used, the features appearing in the nodes of the decision tree can be presented as evidence. Alternatively, if a deep learning algorithm with an attention mechanism is used, the features corresponding to the attention that contributed to the classification can be presented as evidence. When generating classification evidence, the evidence generation unit 430 may also present which part of the classification criterion data 440 the features used as evidence correspond to. This allows the user to know where the presented features correspond to in the classification criterion data 440, and also to refer to surrounding information of the classification criterion data 440, thereby gaining a satisfactory understanding of the presented evidence.

[0029] In Figure 1, the classification criterion data 340 and document classification model 350 held by the classification model learning server 30 are identical to the classification criterion data 440 and document classification model 450 held by the document classification server 40. When the classification model learning server 30 has finished training the document classification model 350, the classification criterion data 340 and document classification model 350 are transmitted from the classification model learning server 30 to the document classification server 40, and these become the classification criterion data 440 and document classification model 450. Therefore, if the classification model learning server 30 and the document classification server 40 are configured on the same computer, the classification criterion data 340 and classification criterion data 440, and the document classification model 350 and document classification model 450 may be integrated into a single entity.

[0030] Each of the above-mentioned devices constituting the document classification system 1, namely the classification client 20, the classification model learning server 30, and the document classification server 40, can be realized by a computer having a processor, main memory, auxiliary storage, a communication interface, and an input / output interface. Specifically, each part of the classification client 20, the classification model learning server 30, and the document classification server 40 (classification request input unit 210, classification result display unit 220, reference feature extraction unit 310, classification model learning unit 320, reference feature extraction unit 410, document classification unit 420, and basis generation unit 430) is realized by the processor of each device reading a program into the main memory and executing it. Furthermore, the document classification training data 330, classification criterion data 340, and document classification model 350 are stored in the auxiliary storage of the classification model learning server 30 (which may be main memory or an externally connected storage device, etc.), and the classification criterion data 440 and document classification model 450 are stored in the auxiliary storage of the document classification server 40 (which may be main memory or an externally connected storage device, etc.). Furthermore, an input / output interface is used for screen display and input reception by the classification request input unit 210 and the classification result display unit 220, and a communication interface is used for communication with other devices.

[0031] Figure 2 shows an example of the configuration of a classification request input screen. The classification request input screen 211 shown in Figure 2 is an example of an initial screen provided to the classification client 20 for the user to input a classification request, and the classification request input unit 210 controls the display and accepts predetermined input operations.

[0032] As shown in Figure 2, the classification request input screen 211 includes a classification target document input area 212 for entering documents to be classified, and a classification instruction button 213 for instructing the execution of document classification for the documents entered in the classification target document input area 212.

[0033] In the classification client 20, when a user operates an input device (keyboard, mouse, etc.) on the classification request input screen 211 displayed on the screen to input a document to be classified into the document input area 212 and presses the classification instruction button 213, the classification request input unit 210 accepts the classification request for the entered document and sends the request to the document classification server 40. After the document classification server 40 performs document classification, the classification client 20 receives the result and the classification result display unit 220 displays the result on the classification result display screen 221 shown in Figure 3.

[0034] Figure 3 shows an example of the configuration of the classification result display screen. The classification result display screen 221 shown in Figure 3 is an example of a screen provided to the classification client 20 to display the results of document classification performed by the document classification server 40 in response to the classification request entered on the classification request input screen 211, and its display is controlled by the classification result display unit 220.

[0035] In Figure 3, the classification result display screen 221 is displayed at the bottom of the classification request input screen 211. The configuration of the classification request input screen 211 is as described in Figure 2. The classification result display screen 221 has a classification result display area 222 where the results of the document classification are displayed, and a classification basis display area 223 where the basis for the document classification is displayed.

[0036] Figure 4 is a sequence diagram showing an example of the processing procedure for document classification by document classification system 1.

[0037] As shown in Figure 4, first, in the classification model learning server 30, the reference feature extraction unit 310 extracts reference features from the document classification training data 330 and the classification reference data 340 (step S101). Next, the classification model learning unit 320 generates a document classification model 350 by performing model learning using the reference features extracted in step S101 (step S102). Then, the classification model learning server 30 sends the classification reference data 340 used in step S101 and the document classification model 350 generated in step S102 to the document classification server 40 (step S103).

[0038] On the other hand, in the classification client 20, when a user inputs a classification request on the classification request input screen 211, the classification request input unit 210 accepts this input (step S104) and sends the classification request to the document classification server 40 (step S105). As described above with reference to Figure 2, on the classification request input screen 211, the input operation for the classification request is completed when the document to be classified is entered in the document to be classified input area 212 and the classification instruction button 213 is pressed.

[0039] Next, in the document classification server 40, the reference feature extraction unit 410 extracts reference features from the classification reference data 440 and the documents (documents to be classified) received from the classification client 20 in step S105 (step S106). Next, the document classification unit 420 classifies the target documents using the reference features extracted in step S106 (step S107) and transmits the classification result to the classification client 20 (step S108). Furthermore, the basis generation unit 430 generates information (classification basis) that serves as the basis for the document classification in step S107 (step S109) and transmits this to the classification client 20 (step S110).

[0040] Finally, in the classification client 20, the classification result display unit 220 generates a classification result display screen 221 based on the information received from the document classification server 40 in steps S108 and S110, and displays it on the display of the classification client 20 (step S111). As described above with reference to Figure 3, in the classification result display screen 221, the classification result received in step S108 is displayed in the classification result display area 222, and the classification basis received in step S110 is displayed in the classification basis display area 223.

[0041] As described above, the document classification system 1 according to this embodiment can perform document classification of target documents specified by the user and display the classification result in the classification result display area 222 on the classification result display screen 221, as well as display the classification basis that forms the basis of the document classification in the classification basis display area 223. The basis generation unit 430 generates features and other elements that contributed to the document classification as classification basis based on the classification criterion data 440 used in the document classification, thereby providing the user with classification basis that is convincing to humans. Thus, according to the document classification system 1 according to this embodiment, the user can accept the results of the document classification with conviction by viewing the classification basis that conforms to the classification criteria for the document classification.

[0042] (2) Second embodiment Figure 5 shows an example of the configuration of the document classification system 2 according to the second embodiment of the present invention. In the description of the second embodiment, the same configuration and processing as the first embodiment will not be repeated in the description.

[0043] As shown in Figure 5, the document classification system 2 according to the second embodiment is configured by connecting a classification client 20 used for inputting classification requests by users and displaying classification results, a classification model learning server 31 for learning a classification model, a document classification server 40 for classifying given documents, and a reference feature setting client 50 that can perform adjustments to the classification reference data 340 generated by the classification model learning server 31 by adding or deleting reference features, via a communication network 10.

[0044] In Figure 5, the classification client 20, classification model learning server 31, document classification server 40, and reference feature setting client 50 are shown as separate devices. However, the document classification system 2 may consist of some or all of these devices on the same computer.

[0045] The classification client 20 and document classification server 40 in document classification system 2 are the same as the classification client 20 and document classification server 40 described in the first embodiment.

[0046] The classification model learning server 31 includes, in addition to the configuration of the classification model learning server 30 in the first embodiment (reference feature extraction unit 310, classification model learning unit 320, document classification training data 330, classification criterion data 340, document classification model 350), a criterion generation unit 360 that generates classification criterion data 340 from the document classification training data 330. That is, in the first embodiment, the classification criterion data 340 was prepared and set in advance, but in the second embodiment, the classification criterion data 340 is generated by the criterion generation unit 360.

[0047] Specifically, the criterion generation unit 360 can generate classification criteria (classification criterion data 340) by extracting words characteristic of the document sets included in each classification category contained in the document classification training data 330. The method for calculating the characteristicness of words in the extraction of characteristic words can be any method, but for example, the publicly known TF-IDF method can be used.

[0048] The reference feature setting client 50 includes a reference feature display unit 510 that displays the classification reference data 340 generated by the reference generation unit 360 of the classification model learning server 31, a reference feature addition unit 520 that adds new reference features to the classification reference data 340 in response to user operations, and a reference feature deletion unit 530 that deletes unnecessary reference features from the classification reference data 340 in response to user operations.

[0049] Figure 6 shows an example of the configuration of the reference feature setting screen. The reference feature setting screen shown in Figure 6 is an example of a screen provided to the reference feature setting client 50 for users to modify the classification reference data 340, in which the reference feature display unit 510 controls the display, and the reference feature addition unit 520 and the reference feature deletion unit 530 accept predetermined input operations.

[0050] The reference feature setting screen shown in Figure 6 is broadly divided into a reference feature display area 511 for displaying and deleting reference features included in the classification reference data 340, and a reference feature addition area 521 for adding reference features.

[0051] In the reference feature display area 511, multiple reference features included in the classification reference data 340 generated by the reference generation unit 360 of the classification model learning server 31 are displayed, each combined with the classification category to which the reference feature belongs. The classification reference data 340 is transmitted from the classification model learning server 31 to the reference feature setting client 50, and the reference feature display unit 510 displays the contents of the received classification reference data 340 in the reference feature display area 511.

[0052] Furthermore, the reference feature display area 511 is provided with delete instruction buttons 531 corresponding to each displayed reference feature. When a user wants to delete any of the reference features displayed in the reference feature display area 511 from the classification reference data 340, they press the delete instruction button 531 corresponding to the reference feature to be deleted. Upon receiving this press operation, the reference feature deletion unit 530 modifies the classification reference data 340 received from the classification model learning server 31 to delete the specified feature, and then sends the modified classification reference data 340 to the classification model learning server 31.

[0053] The base feature addition area 521 includes a feature input area 523 for inputting the feature to be added (a new base feature), a classification category input area 522 for inputting the classification category to which the feature belongs, and a feature addition instruction button 524 for instructing that the feature entered in the feature input area 523 be added to the classification criteria in the classification category entered in the classification category input area 522.

[0054] When adding a feature, the user operates an input device (keyboard, mouse, etc.) to input the desired feature into the feature input area 523 displayed on the screen of the reference feature setting client 50, input the classification category of the feature into the classification category input area 522, and press the feature addition instruction button 524. Upon receiving this press operation, the reference feature addition unit 520 modifies the classification reference data 340 received from the classification model learning server 31 by adding the input feature, and sends the modified classification reference data 340 to the classification model learning server 31.

[0055] Although not shown in Figure 6, the reference feature setting screen may also include a confirmation button that the user can press when they determine that no modifications are needed to the multiple reference features displayed in the reference feature display area 511 (i.e., the classification reference data 340 generated by the reference generation unit 360 of the classification model learning server 31). In this case, when the confirmation button is pressed, the reference feature setting client 50 sends a message to the classification model learning server 31 indicating that no modifications are needed to the classification reference data 340, and the classification model learning server 31 executes subsequent processing without modifying the classification reference data 340. This configuration allows the user to perform a final check of the classification reference data 340 automatically generated by the reference generation unit 360.

[0056] Figure 7 is a sequence diagram showing an example of the processing procedure for document classification by document classification system 2.

[0057] As shown in Figure 7, first, in the classification model learning server 31, the criterion generation unit 360 generates classification criterion data 340 from the document classification training data 330 (step S201), and transmits the generated classification criterion data 340 to the criterion feature setting client 50 (step S202).

[0058] Next, in the standard feature setting client 50, the standard feature display unit 510 displays the standard features included in the classification standard data 340 received in step S202 in the standard feature display area 511 of the standard feature setting screen (step S203). If the user wants to add a feature to the classification standard, as explained in Figure 6, the user performs an input operation for addition in the standard feature addition area 521, and the standard feature addition unit 520 modifies the classification standard data 340 by adding the input feature (step S204). Also, if the user wants to delete a feature from the classification standard, as explained in Figure 6, the user presses the delete instruction button 531 corresponding to the feature to be deleted, and the standard feature deletion unit 530 modifies the classification standard data 340 by deleting the specified feature (step S205). If the classification criterion data 340 is modified in step S204 or step S205, the modified classification criterion data 340 is sent from the criterion feature setting client 50 to the classification model learning server 31 (step S206).

[0059] Then, from step S206 onward, the same processing as the document classification process shown in Figure 4 in the first embodiment (steps S101 to S111) is performed. Although a detailed explanation is omitted, the classification model learning server 31 performs model learning using the reference features extracted from the document classification training data 330 and the modified classification criterion data to generate a document classification model 350 (steps S101 to S103). Then, when the classification client 20 receives input of a target document from a user and requests its classification (steps S104 to S105), the document classification server 40 performs document classification of the target document using the document classification model 350 generated by the classification model learning server 31, generates the classification basis, and sends the classification result and classification basis to the classification client 20 (steps S106 to S110). Then, the classification client 20 generates and displays a classification result display screen 221 based on the received classification result and classification basis (step S111).

[0060] As described above, the document classification system 2 according to this embodiment generates classification criterion data 340 from document classification training data 330, and further modifies the criterion features included in the classification criterion data 340 according to the user's request, and then uses the modified classification criterion data 340 to perform document classification of the target document specified by the user, and can present the classification result and the basis for classification to the user.Therefore, according to the document classification system 2 according to this embodiment, even if there are no (not prepared) classification criteria for document classification, it is possible to generate classification criteria in a way that reflects the user's wishes, perform document classification based on those classification criteria, and output the classification result and the basis for classification in the same way as in the first embodiment, so that the user can accept the results of document classification.

[0061] Furthermore, as a modification of the second embodiment, a configuration can be considered in which the standard feature setting client 50 is excluded from the configuration of the document classification system 2 according to the second embodiment. In a document classification system with such a configuration, the standard generation unit 360 automatically generates classification standard data 340 from the document classification training data 330, and then, without any modifications such as the addition or deletion of features by the user (or even without user confirmation), the generated classification standard data 340 is used by the classification model learning server 31 and the document classification server 40, etc. (steps S101 to S111 in Figure 7). With such a document classification system, even if there are no (not prepared) classification standards for document classification, classification standards can be automatically generated, and based on those classification standards, document classification can be performed in the same way as in the first and second embodiments, and the classification results and classification basis can be output, thus providing the new effect of reducing the effort required for user confirmation. [Explanation of Symbols]

[0062] 1,2 Document Classification System 10 Communication Networks 20 Classification Clients 30,31 Classification Model Training Server 40 Document Classification Server 50. Client for setting reference features 210 Classification Request Input Section 211 Classification Request Input Screen 212 Classification Target Document Input Area 213 Classification instruction button 220 Classification result display section 221 Classification result display screen 222 Classification result display area 223 Classification basis display area 310 Reference Feature Extraction Unit 320 Classification Model Learning Unit 330 Document Classification Training Data 340 Classification Criteria Data 350 Document Classification Models 360 Standard generation section 410 Reference Feature Extraction Unit 420 Document Classification Department 430 Basis Generation Unit 440 Classification Criteria Data 450 Document Classification Models 510 Reference Feature Display Section 511 Reference Feature Display Area 520 Base Feature Addition Section 521 Additional area for standard features 522 Classification Category Input Area 523 Feature Input Area 524 Feature Addition Instruction Button 530 Criteria Feature Removal Unit 531 Delete instruction button

Claims

1. A document classification system for classifying a collection of documents, A classification client that receives requests from users to specify a document set and to perform classification on that document set, A document classification device that classifies a set of documents specified by the classification client, A reference feature setting client that receives instructions from the user regarding the addition or deletion of features to the classification criterion data held by the document classification device, Equipped with, The aforementioned document classification device is A criterion generation unit that generates classification criterion data having multiple features from classification training data, A feature extraction unit extracts the criterion features for classification from the generated classification criterion data, A classification model learning unit that generates a document classification model using the extracted reference features, A document classification unit that classifies the document set using the generated document classification model, The system includes a basis generation unit that generates information representing the basis for the classification of the document set by the document classification unit using the aforementioned standard features, The aforementioned classification client, The system includes a classification result output unit that outputs the classification result of the document collection by the document classification unit and information representing the basis for the classification generated by the basis generation unit. If the addition or deletion of features is specified in the aforementioned feature setting client, the document classification device, specifically the feature extraction unit, modifies the classification criterion data generated by the criterion generation unit according to the specification in the feature setting client, and extracts the classification criterion features using the modified classification criterion data. A document classification system characterized by the following features.

2. The basis generation unit generates information representing the basis for the classification, using the criteria features extracted by the criteria feature extraction unit that contributed to the classification of the document set by the document classification unit. The document classification system according to item 1, characterized in that it is as described above.

3. The aforementioned criterion feature extraction unit extracts classification criterion features using the classification criterion data after the user has confirmed in the criterion feature setting client whether or not modifications to the classification criterion data generated by the criterion generation unit are permissible. The document classification system according to item 2, characterized in that it is as follows:

4. A document classification device for classifying a collection of documents, A criterion generation unit that generates classification criterion data having multiple features from classification training data, A feature extraction unit extracts the criterion features for classification from the generated classification criterion data, A classification model learning unit that generates a document classification model using the extracted reference features, A document classification unit that classifies a set of documents specified by a user using the generated document classification model, A basis generation unit generates information representing the basis for the classification of the document set by the document classification unit using the aforementioned standard features, Equipped with, When the user provides instructions regarding the addition or deletion of features to the classification criterion data generated by the criterion generation unit, the criterion feature extraction unit modifies the classification criterion data generated by the criterion generation unit according to the instructions and extracts classification criterion features using the modified classification criterion data. A document classification device characterized by the following features.

5. A document classification method using a document classification system that classifies a set of documents, The document classification system comprises a classification client that receives requests from users to specify a set of documents and to perform classification on said set of documents; a document classification device that classifies the set of documents specified by the classification client; and a reference feature setting client that receives instructions from the user regarding the addition or deletion of features to the classification reference data held by the document classification device. The classification client includes a classification request input step in which it receives the specification of the document set and a request to perform the classification, The document classification device includes a criterion generation step in which it generates classification criterion data having multiple features from classification training data, The document classification device includes a criterion feature extraction step which extracts criterion features for classification from the classification criterion data generated in the criterion generation step, The document classification device includes a classification model learning step in which it generates a document classification model using the reference features extracted in the reference feature extraction step, The document classification device performs a document classification step in which it classifies a set of documents specified in the classification request input step using the document classification model generated in the classification model learning step, The document classification device includes a basis generation step in which it generates information representing the basis for classifying the document set in the document classification step using the standard feature quantities, The classification client includes a classification result output step which outputs the classification result of the document set by the document classification step and information representing the basis for the classification generated in the basis generation step, Equipped with, If the addition or deletion of features is specified in the aforementioned criterion feature setting client, the document classification device modifies the classification criterion data generated in the criterion generation step according to the specification, and extracts the classification criterion features using the modified classification criterion data. A document classification method characterized by the following features.

Images