Classification support device, classification support system

Abstract

To improve efficiency of formulating a classification assignment range as a logical retrieval expression.SOLUTION: A classification support device takes a document set 1 with a specific classification and a document set 2 without such classification as input, and extracts a keyword candidate that hits as many as possible from a document comprising the document set 1 and as few as possible a document comprising the document set 2 among document text of the document set 1 and the document set 2. The device outputs: document number containing the keyword candidate or its ratio (recall ratio) to the document number comprising the document set 1; document number containing the keyword candidate or the ratio (relevance ratio) of the document number to the document number comprising the document set 2; and a keyword candidate extracted with at least one of the harmonic mean (F-value) of the two types of ratios. In addition, the device generates a search formula by combining the keyword candidates selected by a user.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a classification-assigning support device and a classification-assigning support system.

Background Art

[0002] In recent years, an environment has been created in which a large number of documents can be easily accessed via the Internet or the like. When it is desired to collect a desired document, by accessing a document search system and specifying search keywords or classifications previously assigned to the documents as search conditions, the desired documents can be narrowed down. In particular, by previously assigning classifications according to the format and content of the documents, the efficiency of the work of identifying the desired documents can be significantly improved.

[0003] However, it is not easy to previously assign appropriate classifications to a large number of documents. Generally, the work of having a human read and understand the content of a document and assign an appropriate classification without excess or deficiency requires a great deal of cost. With the development of recent natural language processing and machine learning technologies, it has become possible to automatically assign classifications to documents by a computer, and an economic improvement in such costs can be expected, but the accuracy of such assignment is not yet high enough.

[0004] One of the reasons for the high cost of the classification-assigning work is that it takes time to read and understand the content of the document. Generally, in order to assign a classification, it is necessary to read the document and fully understand its content, but reading a long text of dozens of pages, such as a patent, requires a great deal of time and effort.

[0005] Another reason for the high cost of classification is the ambiguity of classification boundaries. In classification systems such as the Japanese Decimal Classification (NDC) for books and the IPC, FI, F-terms, etc. for patents, the scope of each classification is defined to some extent, but not strictly, so the classification results often differ depending on the person. Furthermore, even when classification is automatically assigned by computer, the accuracy of classification tends to be low for documents located near the classification boundaries because the computer does not understand the meaning of the document.

[0006] One method to eliminate the two causes mentioned above and perform the classification assignment process efficiently is to define (formulate) the scope of each classification as a rule. For example, a rule such as "If the word 'translation device' appears in a document, assign the classification 'machine translation' to this document" can be used. If this rule can be formalized in some form for each classification, the machine can instantly and easily determine whether or not to assign that classification to a new document based on whether or not the rule is met.

[0007] However, because the scope of the original classification is not strictly defined, formulating this rule for each classification is not easy. Considering the intended use of "using classification results to refine document searches," this rule needs to "completely include all documents assigned to that classification and exclude as many documents as possible that are not assigned to that classification." However, it is not easy for humans to come up with such a rule, and rules that meet a certain level of accuracy (without omissions or noise below a certain level) are likely to have a very complex structure and content. Therefore, machine assistance is needed to enable humans to formulate this rule for each classification at low cost.

[0008] One relevant prior art document is Non-Patent Literature 1. This document describes a method for automatically generating search queries that can search only specific document sets without any omissions or excesses. However, because this method generates search queries automatically, it often fails to produce the search query intended by the user.

[0009] Another relevant prior art is Patent Document 1. Patent Document 1 discloses a technique for improving the accuracy of automatic classification assignment by assigning classifications to training documents that have generated classification rules for each classification, selecting documents that have been incorrectly classified, and performing a classification rule improvement process by adding or changing the weights of classification rules. [Prior art documents] [Patent Documents]

[0010] [Patent Document 1] Japanese Patent Publication No. 2002-202984 [Non-patent literature]

[0011] [Non-Patent Document 1] Makoto Iwayama: "Automatic Generation of Boolean Search Expressions from Document Sets," Proceedings of the 18th Annual Meeting of the Association for Natural Language Processing, pp. 1336-1339, 2012 / 3 [Overview of the project] [Problems that the invention aims to solve]

[0012] As mentioned above, in order to efficiently collect desired documents from a large volume of documents, it is effective to pre-classify the documents. To reduce the cost of pre-classifying documents, it is effective to formalize the scope of each classification. However, in order to streamline the process of formalizing the scope of each classification, the challenge is to provide machine support for the classification formalization process.

[0013] Therefore, the present invention aims to provide a device or the like that mechanically assists the above-mentioned formulation process. The present invention is based on the following two points. (1) The scope of each classification is defined by a logical search expression in which keywords are combined with logical operators such as AND / OR / NOT. It is formulated using (which may be referred to as the "classification search formula" below). (2) The classifications to be assigned are defined, and there are documents for which it has been determined whether or not a classification should be assigned. In other words, the present invention aims to provide a device that reduces work costs by assisting humans in creating logical search formulas (classification search formulas) that can accurately and completely search only the documents to which a classification has been assigned, given that existing documents include both documents to which a classification has been assigned and documents to which a classification has not been assigned. [Means for solving the problem]

[0014] According to a first aspect of the present invention, the following classification support device is provided. The classification support device comprises a processing device. The processing device takes a set of documents 1 to which a specific classification has been assigned and a set of documents 2 to which no such classification has been assigned as input, and extracts keyword candidates from the document texts constituting the set of documents 1 and the set of documents 2 that hit as many documents as possible in the set of documents 1 and as few documents as possible in the set of documents 2. The processing device outputs the extracted keyword candidates to an output device along with at least one of the harmonic mean of the number of documents containing the keyword candidates or their ratio to the number of documents constituting the set of documents 1, the number of documents containing the keyword candidates to the number of documents constituting the set of documents 2, or the ratio of the number of sentences containing the keyword candidates to the number of sentences constituting the set of documents 1 and the number of sentences containing the keyword candidates to the number of sentences constituting the set of documents 2. The processing device generates a search expression by combining the keyword candidates selected by the user using an input device.

[0015] According to a second aspect of the present invention, the following classification support system is provided. The classification support system comprises a processing unit, a storage device, an output device, and an input device. The processing unit uses a keyword candidate extraction unit stored in the storage device to take a document set 1 to which a specific classification has been assigned and a document set 2 to which no such classification has been assigned as input, and extracts keyword candidates from the document texts constituting document set 1 and document set 2 that hit as many documents constituting document set 1 as possible and hit as few documents constituting document set 2 as possible. The processing unit uses a screen generation and display unit stored in the storage device to output the extracted keyword candidates to the output device along with at least one of the harmonic mean of the number of documents containing the keyword candidates relative to the number of documents constituting document set 1 or the ratio thereof, the number of documents containing the keyword candidates relative to the number of documents constituting document set 2 or the ratio of the number of sentences containing the keyword candidates relative to the number of sentences constituting document set 1 and the number of sentences containing the keyword candidates relative to the number of sentences constituting document set 2. The processing unit uses a search expression data management unit stored in the storage device to generate a search expression by combining keyword candidates selected by the user using the input device. [Effects of the Invention]

[0016] According to the present invention, a device is provided that assists a human in creating a logical search formula (classification search formula) that can retrieve only the documents assigned a classification, without any discrepancies, from existing documents that contain both documents with and without classifications, thereby reducing work costs. [Brief explanation of the drawing]

[0017] [Figure 1] This figure shows an example of a block diagram of a classification support system to which the present invention is applied. [Figure 2] This figure shows an example of the hardware configuration of a document processing support system to which the present invention is applied. [Figure 3] This figure shows an example of an outline of a work procedure using the document work support system to which the present invention is applied. [Figure 4] Figure showing an example of a display screen (specification of target classification and classified document set) of a document work support system to which the present invention is applied [Figure 5] Figure showing an example of a display screen (presentation of keyword candidates) of a document work support system to which the present invention is applied [Figure 6] Figure showing an example of a display screen (presentation of synonyms of keyword candidates) of a document work support system to which the present invention is applied [Figure 7] Figure showing an example of a display screen (presentation of synonyms of keyword candidates) of a document work support system to which the present invention is applied [Figure 8] Figure showing an example of a display screen (presentation of related words of keyword candidates) of a document work support system to which the present invention is applied [Figure 9] Figure showing an example of a display screen (presentation of related words of keyword candidates) of a document work support system to which the present invention is applied [Figure 10] Figure showing an example of a display screen (presentation of NOT words) of a document work support system to which the present invention is applied [Figure 11] Figure showing an example of a display screen (presentation of NOT words) of a document work support system to which the present invention is applied [Figure 12] Figure showing an example of a display screen (registration of next keyword candidate) of a document work support system to which the present invention is applied [Figure 13] Figure showing an example of a process for specifying keyword candidates of a document work support system to which the present invention is applied

Mode for Carrying Out the Invention

[0018] The present invention relates to a technology for assisting the work of assigning classifications to a large number of documents. Here, as an example of an embodiment of the present invention (embodiment), a classification assistance system for assisting the work of creating a logical search formula that can search without excess or deficiency for sentences related to patents to which the existing patent classification is applied will be described with respect to the existing patent classification.

[0019] In this embodiment, the patent classification to be worked on is 5B091, an existing F-term (theme) corresponding to "machine translation." However, any other F-term may be used, as long as there is a set of documents to which this classification is assigned and a set of documents to which this classification is not assigned. Other patent classification systems such as FI and IPC may also be used, as well as classifications other than patents, and documents other than patents (papers, web pages, etc.) may also be used as target documents.

[0020] This system is a computer system that interactively works with the operator (user) to create a classification search formula that can accurately and completely search for only the patent sets assigned classification 5B091 from both the patent sets assigned classification 5B091 and the patent sets not assigned classification 5B091.

[0021] The classification search formulas created by operators using this system are assumed to be logical search formulas in which multiple keywords are linked by logical operators AND / OR / NOT. Alternatively, instead of AND, proximity search, which is often used in patent searches, may be employed.

[0022] Figure 1 shows an example of a block diagram of a classification support system to which the present invention is applied. The operator inputs data such as target classifications, sets of documents to which the target classification has been assigned (hereinafter sometimes referred to as "classification-assigned document sets"), and keyword candidates via the input / output device 101. The target classifications specified by the operator are stored in the target classification data 102, and the classification-assigned document sets specified by the operator are stored in the classification-assigned document list 103.

[0023] When an operator requests, via the input / output device 101, for keyword candidates suitable for constructing a classification search formula, the search formula data management unit 104 instructs the keyword candidate extraction unit 107 of the keyword candidate extraction processing unit 106 to extract keyword candidates from the set of classified documents stored in the classification document list 103. Here, using the processing procedure described later (exemplified in Figure 13), keywords are identified that have a higher number of hits in the set of classified documents and a lower number of hits in the set of documents that were not classified. These keywords are then presented to the operator on a screen (exemplified in Figure 5), which will be explained later, along with the number of hit documents and their ratio (recall, precision, F-score). Here, the set of documents that were not classified can be obtained by subtracting the set of classified documents stored in the classification document list 103 from the total set of documents stored in the document search system 113.

[0024] Furthermore, if the operator requests the presentation of synonyms for the keyword candidate via the input / output device 101, the search expression data management unit 104 instructs the synonym candidate extraction unit 108 of the keyword candidate extraction processing unit 106 to perform the process of extracting synonyms for the keyword candidate specified by the operator.

[0025] Here, the process of extracting synonyms can be achieved by employing at least one of the following methods: extracting words containing substrings that constitute the keyword candidate from words registered in the word dictionary 112 as synonyms; extracting synonyms by searching an existing thesaurus; extracting keywords linked with the keyword candidate by logical OR in past search expressions stored in the search history 116 of the document search system 113 as synonyms; or extracting synonyms from the similarity between word vectors using recent machine learning techniques such as Word2vec (i.e., a method of extracting synonyms by searching a thesaurus consisting of a model that can be generated based on the similarity between word vectors using known machine learning techniques).

[0026] Furthermore, if the operator requests the presentation of related words for the keyword candidate via the input / output device 101, the search expression data management unit 104 instructs the related word candidate extraction unit 109 of the keyword candidate extraction processing unit 106 to perform the process of extracting related words for the keyword candidate specified by the operator. Here, related words refer to words that co-occur with the keyword candidate. Specifically, this refers to at least one word that forms a compound word when linked with the keyword candidate, a word that has a dependency relationship (subject-verb, object-verb) with the keyword candidate, or a word other than the above that co-occurs in the same sentence or paragraph. These related words can be obtained by the text analysis unit 111 by referring to the word data 115, which stores word data obtained by performing natural language analysis (morphological and syntactic analysis) on the document text 114, which stores the text data of the classification assignment document list 103.

[0027] Furthermore, if an operator requests, via the input / output device 101, to present NOT words for the keyword candidates or their synonyms and related words, the search expression data management unit 104 instructs the NOT word candidate extraction unit 110 of the keyword candidate extraction processing unit 106 to perform the process of extracting NOT words for the keyword candidates or their synonyms and related words specified by the operator. Here, a NOT word is a word that encompasses the keyword candidates or their synonyms and related words, and is considered to have little semantic relevance to the keyword candidates or their synonyms and related words, and is deemed inappropriate to include in the search results (noise keyword). For example, in a general full-text search system, a search keyword "cleaning" may incorrectly hit documents containing "screening," but by specifying a NOT word in the search condition, documents containing "cleaning" can be hit, but documents containing "screening" (noise keyword) can not. These NOT words can be realized by employing at least one of the following methods: searching for headwords in a word dictionary 112 and extracting words containing the keyword candidate or its synonyms and related words as NOT words; or, as illustrated later (as illustrated in Figure 10), scanning a document text 114 containing the keyword candidate or its synonyms and related words, extracting the context before and after the keyword candidate, etc., presenting it to the operator via an input / output device 101, and having the operator select a NOT word.

[0028] The keyword candidate data extracted by the keyword candidate extraction processing unit 106 is incorporated into the screen generated by the screen generation and display unit 117 and presented to the operator via the input / output device 101. Furthermore, the keyword candidate data is stored and managed in the keyword candidate data 105 of the search formula data management unit 104.

[0029] Figure 2 shows an example of the hardware configuration of a classification support system to which the present invention is applied. This system can be configured primarily using a processing unit 230 that performs calculation processing, an input device 210 for users to input operation instructions or data, an output device 220 for outputting calculation processing results to the user, and a storage device 240 that stores programs and data related to the processing performed by the processing unit 230.

[0030] The input device 210 consists of a keyboard 211 and a mouse 212. The output device 220 consists of an output monitor 221. Alternatively, a touch panel or the like that integrating the input and output devices may be used. When exchanging input / output data with another computer (for example, a computer related to the document search system 113), the input / output data is transmitted and received via the network 250.

[0031] The storage device 240 includes a working area 2401 for temporarily storing processing data from the processing device 230. It also includes areas for storing programs, such as a search expression data management unit storage area 2404, a keyword candidate extraction processing unit storage area 2406, a keyword candidate extraction unit storage area 2407, a synonym candidate extraction unit storage area 2408, a related word candidate extraction unit storage area 2409, a NOT word candidate extraction unit storage area 2410, a text analysis unit storage area 2411, a document search system storage area 2413, and a screen generation / display unit storage area 2417. Furthermore, it includes areas for storing data, such as a target classification data storage area 2402, a classification assigned document list storage area 2403, a keyword candidate data storage area 2405, a word dictionary storage area 2412, a document text storage area 2414, a word data storage area 2415, and a search history storage area 2416. The processing unit 230 performs processing by repeatedly loading the necessary programs and data from the storage device 240 and storing the execution results back into the storage device 240. The search expression data management unit storage area 2404 stores the programs used to generate the classification search expressions (search expressions) included in the search expression data management unit 104.

[0032] Figure 3 shows an example of an outline of the procedure for creating a classification search formula using a document work support system to which the present invention is applied. First, the operator registers the target classification data 102 and the classification assignment document list 103 into the system (step 301). Next, the system analyzes the document text in the classification assignment document list 103 in response to the operator's request, extracts keyword candidates, and presents them to the operator. The operator selects an appropriate keyword candidate from the presented candidates (step 302). Next, the system extracts synonym candidates for the keyword candidate selected by the operator in response to the operator's request and presents them to the operator. The operator selects an appropriate synonym candidate from the presented candidates (step 303). Next, the system extracts related word candidates for the keyword candidate selected by the operator in response to the operator's request and presents them to the operator. The operator selects an appropriate related word candidate from the presented related word candidates (step 304). Next, the system extracts data related to the location where NOT word candidates or NOT word candidates appear for the keyword candidate, synonym candidate, and related word candidate selected by the operator, and presents it to the operator. The worker selects the appropriate NOT word candidate by referring to the presented NOT word candidates or location data (step 305).

[0033] Next, the system merges the keyword candidates, synonym candidates, related word candidates, and NOT word candidates selected by the operator and registers them in the system as components of a classification search formula (step 306). Next, the operator determines whether they have created an appropriate classification search formula that can search the classification document list 103 completely and without omissions (step 307). If they determine that they have created an appropriate classification search formula, they end the work (step 308). If they determine that they have not yet created an appropriate classification search formula, the system generates a classification document list 103 in which the classification documents that can be searched with the current classification search formula (components) are excluded from the classification document list 103. Then, using this classification document list 103, they return to step 302 and repeat the same process to update the classification search formula.

[0034] In Figure 3, the process is performed in the following order: presentation and selection of keyword candidates, presentation and selection of synonym candidates, presentation and selection of related word candidates, and presentation and selection of NOT word candidates. However, the order after the presentation and selection of keyword candidates is not particularly prescribed, and the process can be performed in any order.

[0035] Figure 4 shows an example of a display screen of a document work support system to which the present invention is applied, and is a diagram showing an example of a screen for specifying the target classification and the set of documents to which classification is assigned. First, the operator uses screen 401 to input a classification name 402 for the classification search formula to be created. Any name that distinguishes it from other classifications is acceptable. Next, the operator specifies the set of documents to which the classification has been assigned. Pressing the reference button 403 displays a file list, and the operator registers the classification document list 103 by selecting the text file to be stored in the classification document list. Alternatively, pressing the search button 404 calls up the document search system 113, where the operator specifies the classification (5B091 in this embodiment) as the search condition and performs a search, registering the set of documents output as the search results as the classification document list 103. The registered classification document list 103 is displayed in the display area 405. After confirming the displayed classification document list, the operator presses the register button 406 to save the classification document list 103 to the system. The cancel button 407 is used to cancel the classification document list 103 displayed in the display area 405.

[0036] Figure 5 shows an example of the display screen of a document work support system to which the present invention is applied, and is a diagram showing an example of keyword candidate presentation. When the analysis button 502 is pressed on screen 501 shown in Figure 5, the system extracts keyword candidates according to the procedure described later (exemplified in Figure 13) and presents a list of keyword candidates to the operator in table format. This table consists of keyword candidates 503, the number of documents hit in the classification document list 103 504, the number of documents hit in all documents as unclassified texts 505, recall 506 which indicates the degree of accuracy in classification assignment, precision 507 which indicates the degree of noise in classification assignment, and the F-score 508 which represents the harmonic mean of recall and precision. The table in Figure 5 is displayed sorted in descending order by the F-score, but it can be sorted by any column by pressing the ▽ mark next to the item name.

[0037] Here, regarding the definition of a "document that hits" a keyword, in a general document search system 113, if a keyword is included in the document, it is considered a "hit" and included in the search results. However, in this embodiment, the determination is whether or not a document should be assigned a classification. Therefore, for example, it is doubtful whether a document in which the keyword "translation" appears only once should be assigned the classification "machine translation". If it is truly an invention related to machine translation, then "translation" (or a synonym equivalent) should appear multiple times in the document. Therefore, in this embodiment, instead of simply considering the number of documents found by the search system as a "hit document," a post-processing step may be added to recognize only documents in which the frequency of occurrence within each document is above a predetermined threshold as "hit documents". This post-processing can exclude, to some extent, noisy documents to which the classification should not be assigned.

[0038] The worker selects one keyword candidate that they deem appropriate from the presented keyword candidates. The criteria for this selection are as follows: The word "language" shown in Figure 5 hits 1,171 out of 3,139 classified documents, indicating relatively few classification omissions (relatively high recall (recall = 1,171 ÷ 3,139 = 37.3%)). However, the total number of hits is 8,455, indicating relatively high classification noise (relatively low precision (precision = 1,171 ÷ 8,455 = 20.1%)). On the other hand, the word "translation" hits 717 out of 3,139 classified documents, indicating relatively few classification omissions (relatively high recall (recall = 717 ÷ 3,139 = 22.8%)). On the other hand, the total number of hits is 3,578, and the classification noise is less than that of "language" (precision = 717 ÷ 3,578 = 20.0%). Generally, it is difficult to determine at this stage which word is appropriate to select as a keyword candidate. By selecting the word "language" and connecting it with other related words using logical AND, it may be possible to create an appropriate classification search formula with reduced classification noise. Alternatively, by selecting the word "translation" and connecting it with other synonyms using logical OR, it may be possible to create an appropriate classification search formula with improved classification omissions. In this embodiment, considering the data on the number of hits and the fact that the target classification is "machine translation," it is considered more appropriate to use the word "translation," which specifies the field of machine translation, than to use "language," which is a general word in the field of language processing. Furthermore, selecting keyword candidates from words with relatively high F-scores is also considered an effective indicator. In this embodiment, we will proceed with the next steps using "translation" as the keyword candidate. Pressing the synonym button 509 in Figure 5 will take you to a screen where you can select synonyms for the keyword candidate "translation" (as illustrated in Figure 6 below).

[0039] Figures 6 and 7 show an example of the display screen of a document work support system to which the present invention is applied, and illustrate an example of the presentation of synonyms for keyword candidates. In the screen 601 shown in Figure 6, when the keyword candidate "translation" is selected from the candidates 603 and the synonym button 609 is pressed, the synonyms are presented to the operator in a table format at the bottom of the screen. The symbols (604-608) are the same as in Figure 5. Here, as with the keyword candidates, it consists of the keyword candidate synonyms 613, the number of documents that hit within the classification document list 614, the number of documents that hit within all documents as unclassified texts 615, recall 616, precision 617, and F-score 618. As mentioned above, synonym extraction can be achieved by employing at least one of the following methods: extracting words containing substrings that constitute the keyword candidate as synonyms from words registered in the word dictionary 112; extracting synonyms by searching an existing synonym dictionary; extracting keywords linked with the keyword candidate by logical OR in past search expressions stored in the search history 116 of the document search system 113 as synonyms; or extracting synonyms from the similarity between word vectors using recent machine learning techniques such as Word2vec. The operator selects an appropriate word as a synonym for the keyword candidate "translation" from the presented synonym candidates, taking into account the meaning of the synonyms, the data on the number of hits for the synonyms, and the fact that the target classification is "machine translation".

[0040] Figure 7 shows the screen that appears after the user selects "parallel translation," "interpretation," "Japanese translation," "English translation," and "literal translation" as synonyms for the keyword candidate "translation" in Figure 6 and presses the link button 619. The symbols (703-708, 713-718, 719) are the same as in Figure 6. As shown in the table at the bottom of screen 701 in Figure 7, the selected synonyms are linked to the keyword candidate "translation" using a logical OR. In addition, the number of documents that hit within the classification-assigned document list (724), the number of documents that hit within all documents as unclassified texts (725), recall (726), precision (727), and F-score (728) are calculated for this logical OR, and the calculation results are presented to the user. It can be seen that by linking synonyms to the keyword candidate using OR, the recall improves from 22.8% to 24.0%, and the precision also improves from 20.0% to 20.5%. By checking these values, the operator can confirm that the classification accuracy of the classification search formula they have created is gradually improving. If the operator recalls a synonym not displayed on the screen, they can add it to the classification search formula element 723. Then, by pressing the recalculation button 729, the hit count 724, total hit count 725, recall rate 726, precision 727, and F-score 728 are recalculated, updated, and displayed after the additional input.

[0041] Figures 8 and 9 show examples of display screens of a document work support system to which the present invention is applied, and illustrate an example of the presentation of related words for keyword candidates. Figure 8 shows the screen that transitions to after pressing the related word button 730 in Figure 7. Note that the reference numerals (803 to 808) are the same as in Figure 7. In the screen 801 shown in Figure 8, when "Translation or Parallel Translation or Interpretation or Japanese Translation or English Translation or Literal Translation" is selected from the classification search formula elements 803 and the compound button 809 is pressed, the words that make up the compound word formed by linking "Translation or Parallel Translation or Interpretation or Japanese Translation or English Translation or Literal Translation" are presented as related word candidates in compound word candidate 813 (candidate 813 in Figure 8). Here, xx in Figure 8 refers to "Translation or Parallel Translation or Interpretation or Japanese Translation or English Translation or Literal Translation" for the sake of drawing size. Similar to keyword candidates and synonyms, the hit count 814, total hit count 815, recall rate 816, precision rate 817, and F-score 818 are calculated and presented. The operator refers to these values ​​and selects compound word candidates that seem likely to effectively narrow down the documents that hit the keyword candidate "Translation" and its synonyms.

[0042] Figure 9 shows the screen that appears after the operator selects "xx device," "xx system," "xx method," "machine xx," and "automatic xx" as compound words in Figure 8 and presses the link button 819. The symbols (903-908, 913-918, 922) are the same as in Figure 8. In screen 901 shown in Figure 9, the selected compound words are linked using logical OR. Similarly, for this logical OR, the number of hits (924), total hits (925), recall (926), precision (927), and F-score (928) are calculated and the results are presented to the operator. It can be seen that by compounding keyword candidates and synonyms, the recall decreases from 24.0% to 13.2%, but the precision improves significantly from 20.5% to 66.0%. By checking these values, the operator can confirm that the classification accuracy of the classification search formula they have created is gradually improving. If the operator recalls keyword candidates not displayed on the screen, they can add them to the classification search formula element 923. Then, by pressing the recalculate button 929, the hit count 924, total hit count 925, recall rate 926, precision 927, and F-score 928 are recalculated, updated, and displayed after the additional input.

[0043] Figures 8 and 9 show an example of a screen when compound words are treated as related words, but dependent words and co-occurring words can also be treated as related words instead of compound words. In the case of dependent words, when the dependency button 810 in Figure 8 is pressed, words that have a subject-predicate or object-predicate relationship with the keyword candidate "translate" are extracted and presented using syntactic analysis, which is a well-known method in natural language processing. For example, if there is a sentence "The computer translates the text", the subject "computer" and object "text" of the predicate "translate" are extracted, and they are linked with the keyword candidate "translate" using AND, and presented to the operator as a dependent word candidate 813 (candidate 813 in Figure 8). In the case of co-occurring words, when the co-occurring button 811 in Figure 8 is pressed, words (nouns) that co-occur with the keyword candidate "translate" in the same sentence are extracted and presented using morphological analysis, which is a well-known method in natural language processing. For example, if there is a sentence like "The computer translates the text into English," the words "computer," "text," and "English" that co-occur with the keyword candidate "translate" are extracted, and these are linked with the keyword candidate "translate" using AND, and presented to the worker as a co-occurring word candidate 813 (candidate 813 in Figure 8).

[0044] Figures 10 and 11 show examples of display screens of a document work support system to which the present invention is applied, and illustrate an example of the presentation of NOT words such as keyword candidates. Figure 10 shows the screen that transitions to after pressing the NOT word button 931 in Figure 9. In the screen 1001 shown in Figure 10, the symbols (1004-1008) represent data from the classification search formula element 1003. When "translation device" is selected from the classification search formula element 1003 on screen 1001 and the NOT word button 1009 is pressed, words containing "translation device" but unrelated to language translation are presented as NOT word candidates 1010. The presented word "cell translation device" is extracted from the words registered in the word dictionary 112 that contain "translation device". In other words, "cell translation device" is a word identified as a candidate for noise keyword. The operator checks the presented words and selects words unrelated to language translation as NOT words. In this case, "cell translation device" is unrelated to language translation, so it is selected as a NOT word. For NOT words not registered in the word dictionary 112, the strings before and after the location where "translation device" is written are presented as contextual information 1013. At this time, by presenting the "translation device" in the center and sorting by the strings before and after it (by pressing Sort Before 1011 and Sort After 1012), sections with the same context can be grouped together, allowing for faster identification of NOT words.

[0045] Figure 11 shows the screen that appears after the operator selects "cell translation device" and "in vitro translation device" as NOT words in Figure 10 and presses the link button 1014. Note that the codes (1103-1108, 1110-1114) on screen 1101 are the same as in Figure 10. These NOT words are linked to "translation device" using the NOT operator and presented in the classification search formula element 1123. Additionally, the number of hits (1124), the total number of hits in all documents that did not receive a classification (1125), the recall rate (1126), the precision (1127), and the F-score (1128) are calculated and presented to the operator. It can be seen that the precision has improved from 66.0% to 77.7% by adding the NOT words. By checking these values, the operator can confirm that the classification accuracy of the classification search formula they are creating is gradually improving.

[0046] Figure 12 is a diagram showing an example of a display screen of a document work support system to which the present invention is applied, and is a diagram showing an example of a screen that presents the following keyword candidates. Figure 12 shows the screen that transitions to after the worker presses the registration button 1130 in Figure 11. The classification search formula element 1203, along with its hit count 1204, total hit count 1205, recall rate 1206, precision rate 1207, and F-score 1208 data displayed at the top of screen 1201 shown in Figure 12, are stored in the keyword candidate data 105 of the search formula data management unit 104. The classification search formula element 1203 is one of the elements that make up the classification search formula. Next, when the analysis button 1212 is pressed, a set of 2,728 documents is generated by excluding the 411 documents that are hit by this classification search formula element 1203 from the 3,139 documents assigned the classification "machine translation". This set of documents is used as a new classification-assigned document list 103, and keyword candidates are similarly extracted and presented as keyword candidates 1213. The hit count 1214, total hit count 1215, recall rate 1216, precision rate 1217, and F-score 1218 are also presented. Subsequently, by repeating the steps shown in Figure 5 and beyond, classification search expression elements 1203 are created and added, and finally, these classification search expression elements 1203 are concatenated with OR to create the final classification search expression.

[0047] When the synonym button (509, 609, 709, 1219) is pressed, the user is redirected to a screen related to synonyms. When the related word button (510, 610, 710, 730, 930, 1221) is pressed, the user is redirected to a screen related to related words. When the NOT word button (511, 611, 711, 731, 931, 1109, 1129, 1220) is pressed, the user is redirected to a screen related to NOT words. Additionally, when the compound word button (809, 909, 919) is pressed, the user is redirected to a screen related to compound word candidates. When the dependency button (810, 910, 920) is pressed, the user is redirected to a screen related to dependency words. When the co-occurrence button (811, 911, 921) is pressed, the user is redirected to a screen related to co-occurring words. Then, when the registration button (512, 612, 712, 732, 932, 1130, 1222) is pressed, the current classification search formula (or its components) is registered in the system. Therefore, by using these buttons, the user can transition to the appropriate screen and proceed with the creation of the classification search formula efficiently.

[0048] Referring to Figure 13, an example of the process for extracting keyword candidates will be explained. The keyword candidates are the data exemplified in Figure 5. Figure 13 is a diagram showing an example of the process for identifying keyword candidates in a document work support system to which the present invention is applied. The text analysis unit 111 performs morphological and syntactic analysis on the document text 114, divides the text into words, and analyzes the dependencies between words (step 1301). Next, it extracts nouns from the divided words (step 1302). Steps 1301 and 1302 can be processed in advance, and the results can be stored in the word data 115 to reduce processing time. Next, for each of the classified document set and the document set other than the classified document set in the classified document list 103, the frequency of word occurrences within the documents is calculated, and the number of documents whose frequency of occurrence within the documents is above a threshold is counted (step 1303). Next, for each word, the classification accuracy (recall, precision, F-score) is calculated, and the documents are sorted in descending order by F-score (step 1304).

[0049] Based on the above description, the present invention (apparatus, system) includes means for extracting keyword candidates from the document texts constituting document set 1 and document set 2, taking document set 1 to which a specific classification has been assigned and document set 2 to which no specific classification has been assigned as input, such that the keyword candidates hit as many documents constituting document set 1 as possible and hit as few documents constituting document set 2 as possible; the number of documents (hits) or the ratio (recall rate) of documents containing the keyword candidates to the number of documents constituting document set 1; the number of documents (total hits) to which the keyword candidates have been assigned to the number of documents constituting document set 2; or the ratio (precision rate) of documents (hits) to which documents (total hits) have been assigned to the number of documents constituting document set 1 and documents (total hits) to which documents (total hits) have been assigned to the number of documents constituting document set 2; and means for presenting the extracted keyword candidates to the user along with at least one of the harmonic mean of the two types of ratios; and means for generating a search expression by combining the keyword candidates selected by the user. Furthermore, in a more specific example, the invention provides means for extracting synonym candidates, related word candidates, and noise keyword candidates for keyword candidates selected by the user; the number of documents (number of hits) or their ratio (recall rate) that contain the synonym candidates, related word candidates, and noise keyword candidates to the number of documents constituting the document set 1; the number of documents (total hits) or their ratio to the number of documents (precision rate) that contain the synonym candidates, related word candidates, and noise keyword candidates to the number of documents constituting the document set 2; and means for presenting the extracted synonym candidates, related word candidates, and noise keyword candidates to the user along with at least one of the harmonic mean of the two types of ratios; and means for generating a search expression by combining the synonym candidates, related word candidates, and noise keyword candidates selected by the user. Furthermore, an invention is provided of a method by which a processing device performs these means.

[0050] According to the present invention, it is possible to support the process by which a human can create a logical search expression (classification search expression) that can accurately and completely search only the documents to which a classification has been assigned, given that existing documents contain both documents to which a classification has been assigned and documents to which a classification has not been assigned. Specifically, classification search expressions that define the scope of each classification can be created through trial and error and efficiently while checking the accuracy of the classification assignment by the classification search expression. Furthermore, since keyword candidates that make up the classification search expression can be selected while checking the accuracy of the classification assignment, it is possible to create a classification search expression that takes into account the balance between classification assignment omissions and classification assignment noise according to the purpose of the classification, for example, "I want to apply the classification assignment results to a prior art search for patents, so a certain amount of classification assignment noise is acceptable, but I want to prevent classification assignment omissions as much as possible." In addition, since the scope of the classification can be defined by the logical search expression, when using that classification as a search condition in document search, it is possible to modify and use a part of the classification search expression corresponding to that classification, enabling a search that better matches the search purpose. Furthermore, if it becomes necessary to change the scope of a classification in the future, such as dividing it into multiple subclassifications, it is possible to create classification search formulas corresponding to each subclassification based on the existing classification search formula, thereby reducing the cost of creating classification search formulas.

[0051] Furthermore, the present invention provides a means for excluding documents from document set 1 that are hit by the search expression selected and confirmed by the user, and repeating the above process on the remaining document set 1, thereby creating a final classification search expression.

[0052] Non-patent document 1 mentioned above describes a method for automatically generating search queries that can search only specific document sets without any omissions or excesses. However, because this method generates search queries automatically, it often fails to produce the search queries intended by the human user. In contrast, the present invention provides a work environment in which humans and computers can interactively create search queries, and supports the efficient and highly accurate creation of search queries intended by the human user, taking into account the purpose of classification.

[0053] The aforementioned Patent Document 1 discloses a technique for improving the accuracy of automatic classification assignment by assigning classifications to training documents that have generated classification rules for each classification category, selecting documents that have been incorrectly classified, and performing a classification rule improvement process by adding or changing the weights of classification rules. However, while the prior art (Patent Document 1) improves rules based on mechanically generated rules, the present invention supports the work of creating rules (search expressions) from scratch. Furthermore, the prior art deals with rules based on the presence or absence of words and weighting, while the present invention deals with the creation of logical search expressions (classification search expressions), and there are differences in the generation procedure and support method. In addition, since the present invention creates rules as logical search expressions (classification search expressions), there is an advantage that these rules (classification search expressions) can be embedded in the search conditions created during a search, and parts of them can be added, modified, or deleted for use. Furthermore, while the prior art makes it difficult to optimize the overall weighting of words (a trade-off may occur where increasing the weight of word A results in an inappropriate weighting of word B), the present invention differs in that it allows for optimization of each element constituting the classification search formula.

[0054] Although embodiments of the present invention have been described in detail above, the present invention is not limited to the embodiments described above, and various design modifications can be made without departing from the spirit of the invention as described in the claims. For example, the embodiments described above are described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Moreover, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations.

[0055] The processing unit 230 only needs to function as a processor and can be configured using an appropriate semiconductor device such as a CPU (Central Processing Unit). The storage device 240 can be configured using an appropriate device such as an HDD (Hard Disk Drive). RAM (Random Access Memory) may also be used. Furthermore, the configuration may be modified as appropriate within the scope in which the functions of the present invention are exhibited. For example, the processing unit 230 and the storage device 240 may be composed of one device or multiple devices. Also, the processing unit 230 and the storage device 240 may be composed of devices of the same type or of different types.

[0056] The display method described above is merely an example, and the display method may be changed as appropriate. [Explanation of Symbols]

[0057] 101... Input / Output Device, 102... Target Classification Data, 103... Classification Assigned Document List, 104... Search Formula Data Management Unit, 105... Keyword Candidate Data, 106... Keyword Candidate Extraction Processing Unit, 107... Keyword Candidate Extraction Unit, 108... Synonym Candidate Extraction Unit, 109... Related Word Candidate Extraction Unit, 110... NOT Word Candidate Extraction Unit, 111... Text Analysis Unit, 112... Word Dictionary, 113... Document Search System, 114... Document Text, 115... Word Data, 116... Search History, 117... Screen Generation / Display Unit

Claims

1. Equipped with a processing device, The aforementioned processing apparatus is Taking a set of documents 1 assigned a specific classification and a set of documents 2 not assigned a specific classification as input, the document text constituting each set of documents is divided into words, nouns are extracted from the divided words, and for each extracted word, the hit count, total hit count, recall, precision, and F-score are calculated. The aforementioned hit count is the value of the number of documents that contain the extracted word among the documents that make up the document set 1. The total number of hits is the number of documents containing the extracted word among the documents that make up the document set 2. The recall rate is a value based on the number of documents containing the extracted word among the documents comprising the document set 1 divided by the total number of documents comprising the document set 1. The aforementioned precision is a value based on the number of documents containing the extracted word among the documents constituting the document set 1 divided by the number of documents containing the extracted word among the documents constituting the document set 2. The F-value is the harmonic mean of the recall and the precision. The aforementioned processing apparatus is Among the extracted words, keyword candidates are those whose F-score is equal to or greater than a predetermined value, The output device outputs the number of hits, the total number of hits, the recall rate, the precision rate, and at least one of the F-scores for all words included in the keyword candidates. Using an input device, a search expression is generated by combining keyword candidates selected by the user. The aforementioned processing apparatus is In calculating the number of hits, the frequency of word occurrences within documents is calculated, and documents in which the frequency of word occurrences within documents is above a threshold are counted as the number of documents. In calculating the total number of hits, the frequency of word occurrences within documents is calculated, and documents where the frequency of word occurrences within documents is above a threshold are counted as the number of documents. A classification assignment support device characterized by the following features.

2. A classification support device according to claim 1, The aforementioned processing apparatus is Extract each word that is a synonym for the keyword candidate word selected by the user, The extracted synonym candidate words, along with at least one of the following pieces of information—the number of hits for each word, the total number of hits, the recall rate, the precision rate, and the F-score—are output to the output device. The aforementioned hit count is the number of documents containing the extracted word among the documents comprising the document set 1, and is calculated for each extracted word. The total hit count is the number of documents containing the extracted word among the documents comprising the document set 2, and is calculated for each extracted word. The recall rate is a value based on the number of documents containing the extracted word among the documents comprising the document set 1 divided by the total number of documents comprising the document set 1, and is calculated for each extracted word. The aforementioned precision is a value based on the number of documents containing the extracted word among the documents comprising document set 1 divided by the number of documents containing the extracted word among the documents comprising document set 2, and is calculated for each extracted word. The F-score is the harmonic mean of the recall and precision, and is calculated for each extracted word. The aforementioned processing apparatus is The input device is used to generate a search expression by combining synonym candidate words selected by the user. A classification assignment support device characterized by the following features.

3. A classification support device according to claim 1, The aforementioned processing apparatus is Extract each word that is a related word candidate to the keyword candidate word selected by the user, The extracted related word candidates, along with at least one of the following pieces of information—the number of hits for each word, the total number of hits, the recall rate, the precision rate, and the F-score—are output to the output device. The aforementioned hit count is the number of documents containing the extracted word among the documents comprising the document set 1, and is calculated for each extracted word. The total hit count is the number of documents containing the extracted word among the documents comprising the document set 2, and is calculated for each extracted word. The recall rate is a value based on the number of documents containing the extracted word among the documents comprising the document set 1 divided by the total number of documents comprising the document set 1, and is calculated for each extracted word. The aforementioned precision is a value based on the number of documents containing the extracted word among the documents comprising document set 1 divided by the number of documents containing the extracted word among the documents comprising document set 2, and is calculated for each extracted word. The F-score is the harmonic mean of the recall and precision, and is calculated for each extracted word. The aforementioned processing apparatus is The input device is used to generate a search expression by combining related word candidates selected by the user. A classification assignment support device characterized by the following features.

4. A classification support device according to claim 1, The aforementioned processing apparatus is The system identifies noise keyword candidates that include words selected by the user and are inappropriate to include in search results. The words included in the identified noise keyword candidates are output to the output device. Using the aforementioned input device, a search expression is generated by combining the noise keyword candidate words selected by the user. A classification assignment support device characterized by the following features.

5. A classification assignment support device according to claim 2, The aforementioned processing apparatus is The system identifies noise keyword candidates that include the synonym candidates selected by the user and are inappropriate to include in the search results. The words included in the identified noise keyword candidates are output to the output device. Using the aforementioned input device, a search expression is generated by combining the noise keyword candidate words selected by the user. A classification assignment support device characterized by the following features.

6. A classification support device according to claim 3, The aforementioned processing apparatus is The system identifies noise keyword candidates that include the related word candidates selected by the user and are inappropriate to include in search results. The words included in the identified noise keyword candidates are output to the output device. Using the aforementioned input device, a search expression is generated by combining the noise keyword candidate words selected by the user. A classification assignment support device characterized by the following features.

7. A classification support device according to claim 1, The aforementioned processing apparatus is When presenting the keyword candidates to the user, the words included in the keyword candidates are sorted according to the magnitude of the F value and output to the output device. A classification assignment support device characterized by the following features.

8. A classification support device according to claim 1, The aforementioned processing apparatus is Each time the search expression is updated, the number of hits, the total number of hits, the recall rate, the precision rate, and the F-value are calculated and the results are output to the output device. A classification assignment support device characterized by the following features.

9. A classification assignment support device according to claim 2, The aforementioned processing apparatus is Each time the search expression is updated, the number of hits, the total number of hits, the recall rate, the precision rate, and the F-value are calculated and the results are output to the output device. A classification assignment support device characterized by the following features.

10. A classification support device according to claim 3, The aforementioned processing apparatus is Each time the search expression is updated, the number of hits, the total number of hits, the recall rate, the precision rate, and the F-value are calculated and the results are output to the output device. A classification assignment support device characterized by the following features.

11. A classification support device according to claim 4, The aforementioned processing apparatus is Each time the search expression is updated, the number of hits, the total number of hits, the recall rate, the precision rate, and the F-value are calculated and the results are output to the output device. A classification assignment support device characterized by the following features.

12. A classification assignment support device according to claim 2, The aforementioned processing apparatus is When extracting the aforementioned synonym candidates, at least one of the following is referenced: the search query history used in past document searches, an existing thesaurus, or a thesaurus generated by machine learning based on the similarity between word vectors. A classification assignment support device characterized by the following features.

13. A classification support device according to claim 1, The aforementioned processing apparatus is The documents in document set 1 that match the search formula determined by the user are excluded, and the remaining documents in document set 1 are used to extract candidate keywords, and the extracted candidate keywords are output to the output device. A classification assignment support device characterized by the following features.

14. Processing device and Memory device and Output device and Input device and Equipped with, The aforementioned processing apparatus is Using the keyword candidate extraction unit stored in the aforementioned storage device, Taking a set of documents 1 assigned a specific classification and a set of documents 2 not assigned a specific classification as input, the document text constituting each set of documents is divided into words, nouns are extracted from the divided words, and for each extracted word, the hit count, total hit count, recall, precision, and F-score are calculated. The aforementioned hit count is the value of the number of documents that contain the extracted word among the documents that make up the document set 1. The total number of hits is the number of documents containing the extracted word among the documents that make up the document set 2. The recall rate is a value based on the number of documents containing the extracted word among the documents comprising the document set 1 divided by the total number of documents comprising the document set 1. The aforementioned precision is a value based on the number of documents containing the extracted word among the documents constituting the document set 1 divided by the number of documents containing the extracted word among the documents constituting the document set 2. The F-value is the harmonic mean of the recall and the precision. From the extracted words, those with an F-score of or higher than a predetermined value are selected as keyword candidates. Using the screen generation and display unit stored in the aforementioned storage device, The extracted keyword candidates and, The output device outputs to the output device the number of hits, the total number of hits, the recall rate, the precision rate, and at least one of the F-scores for all words included in the keyword candidates. Using the search expression data management unit stored in the aforementioned storage device, Using the aforementioned input device, a search expression is generated by combining the keyword candidates selected by the user. The aforementioned processing apparatus is In calculating the number of hits, the frequency of word occurrences within documents is calculated, and documents in which the frequency of word occurrences within documents is above a threshold are counted as the number of documents. In calculating the total number of hits, the frequency of word occurrences within documents is calculated, and documents where the frequency of word occurrences within documents is above a threshold are counted as the number of documents. A classification assignment support system characterized by the following features.

Images