Document processing system, document processing method, and program

Abstract

To generate a document vector which reflects an actual importance more than before.SOLUTION: A document processing system 100 manages documents. A morpheme analysis section 102 extracts words from an object document. An appearance frequence calculation section 104 calculates an appearance frequency at which each word extracted by the morpheme analysis section 102 appears in an object document. An importance calculation section 106 calculates an importance of each word, on the basis of the calculated appearance frequency. An importance correction section 107 corrects the calculated importance, on the basis of an item to which each extracted word belongs in the object document. A document vector generation section 109 generates a document vector of an object document, on the basis of the importance of each corrected word.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a document processing system, a document processing method, and a program. [Background technology]

[0002] When natural language processing such as document search, classification, question answering, and machine summarization is performed on documents, document vectors that quantitatively represent the contents of the document are used. Known methods for generating document vectors include the TF-IDF (Term Frequency - Inverse Document Frequency) method and the Bag of Words method. Specifically, these methods generate document vectors by evaluating the importance of individual words extracted from a document based on the frequency of appearance of each word. Patent Document 1 also describes a method for generating document vectors that reflects the dependency structure between words that appear in a document. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2018-136760 Summary of the Invention [Problem to be solved by the invention]

[0004] With conventional document vector generation methods, even if a word is important in a document, if its frequency of occurrence is low it is evaluated as unimportant, meaning that it cannot be said that document vectors that reflect actual importance are generated.

[0005] The present disclosure has been made in consideration of the above-mentioned situation, and aims to provide a document processing system, a document processing method, and a program that can generate document vectors that reflect actual importance more closely than conventional methods. [Means for solving the problem]

[0006] In order to achieve the above object, the document processing system according to the present disclosure includes a word extraction unit, an occurrence frequency calculation unit, an importance calculation unit, an importance correction unit, and a document vector generation unit. The word extraction unit extracts words from a target document. The occurrence frequency calculation unit calculates the occurrence frequency of each word extracted by the word extraction unit in the target document. The importance calculation unit calculates the importance of each word extracted by the word extraction unit based on the calculated occurrence frequency. The importance correction unit corrects the importance calculated by the importance calculation unit based on the item in the target document to which each word extracted by the word extraction unit belongs. The document vector generation unit generates a document vector for the target document based on the importance of each word corrected by the importance correction unit. [Effects of the Invention]

[0007] According to the present disclosure, the importance calculated from the frequency of occurrence is corrected based on the item in the target document to which each word extracted from the target document belongs, making it possible to generate document vectors that more closely reflect actual importance than conventional methods. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a functional configuration diagram of a document processing system according to an embodiment of the present invention; [Figure 2] Diagram showing an example document [Figure 3] FIG. 10 is a diagram showing an example of the configuration of an item importance table according to an embodiment; [Figure 4] 1 is a hardware configuration diagram of a document processing system according to an embodiment of the present invention; [Figure 5] Flowchart of search processing according to an embodiment [Figure 6] Flowchart of document vector generation processing according to an embodiment [Figure 7] FIG. 10 is a diagram showing an example of the configuration of a table stored in a word storage unit before the importance is corrected. [Figure 8] FIG. 10 is a diagram showing an example of the configuration of a table stored in a word storage unit after the importance has been corrected. [Figure 9]Figure showing an example of a search result output screen DETAILED DESCRIPTION OF THE INVENTION

[0009] (Embodiment) Hereinafter, a document processing system 100, a document processing method, and a program according to an embodiment of the present disclosure will be described in detail with reference to the drawings. Note that the same or corresponding parts in the drawings are designated by the same reference numerals.

[0010] FIG. 1 is a block diagram showing a functional configuration of a document processing system 100 according to an embodiment of the present disclosure. The document processing system 100 is a system for managing documents, and specifically, is a computer such as a personal computer (PC) or a server that generates document vectors representing the contents of each document and searches for documents using the generated document vectors. Note that the document processing system 100 may be configured using multiple computers, a cloud, or the like. As shown in FIG. 1, the document processing system 100 includes a document acquisition unit 101, a morphological analysis unit 102, a word storage unit 103, an occurrence frequency calculation unit 104, an occurrence document count calculation unit 105, an importance calculation unit 106, an importance correction unit 107, an item-specific importance table 108, a document vector generation unit 109, a document vector storage unit 110, and a search unit 111.

[0011] The document acquisition unit 101 acquires a document set, which is a set of documents to be searched, from a document database (document DB in the drawing) 20 via a network 10 such as the Internet or a LAN (Local Area Network). The data of the documents stored in the document database 20 is structured, and each document contains one or more items. Words in a document belong to one of the items. The document processing system 100 can also recognize each item in a document. FIG. 2 shows document A as an example of a document stored in the document database 20. Document A contains items such as a header, title, summary, heading, and body text.

[0012] Returning to FIG. 1 , the morphological analysis unit 102 performs morphological analysis on the document represented by the document data acquired by the document acquisition unit 101 to extract words constituting the document. For the morphological analysis here, well-known morphological analyzers such as MeCab (http: / / taku910.github.io / mecab / ) and KyTea (http: / / www.phontron.com / kytea / ) can be used. Then, the morphological analysis unit 102 registers the extracted words for each document in the word storage unit 103. The morphological analysis unit 102 is an example of a word extraction unit of the present disclosure.

[0013] The occurrence frequency calculation unit 104 refers to the word storage unit 103 and calculates the occurrence frequency of each word extracted from the document to be processed. Specifically, the occurrence frequency calculated by the occurrence frequency calculation unit 104 is the number of times each word appears in the document to be processed. Note that the occurrence frequency may be the ratio of the number of times each word appears to the total number of words in the document, and various methods can be used to calculate the word occurrence frequency.

[0014] The document appearance count calculation unit 105 refers to the word storage unit 103 and calculates, for each word identified in the document to be processed, the number of documents in which that word appears in the document set.

[0015] The importance calculation unit 106 calculates the importance of each word extracted from the document to be processed using the TF-IDF method, based on the frequency of appearance of the word calculated by the appearance frequency calculation unit 104 and the number of documents in which the word appears calculated by the number-of-documents appearance calculation unit 105. Specifically, the importance calculation unit 106 calculates the importance using the following formula (1). (1) W=m*log(N / n+1)

[0016] In the above formula (1), W represents the importance of a word. m represents the frequency of occurrence of the word calculated by the frequency of occurrence calculation unit 104. N represents the number of documents constituting the document set acquired by the document acquisition unit 101. n represents the number of documents in which the word appears calculated by the document occurrence count calculation unit 105. Note that "log" in formula (1) means common logarithm. From formula (1), it can be seen that the greater the frequency m of occurrence of a word in a document, the greater the importance of the word. It can also be seen that the fewer the number n of documents in which a word appears in the document set, the greater the importance of the word.

[0017] Note that the above formula (1) is an example, and the importance calculation unit 106 may calculate the importance of a word from the frequency of appearance of the word and the number of documents in which the word appears, using other formulas.

[0018] The importance correction unit 107 corrects the importance of each word stored in the word storage unit 103 by referring to the item importance table 108 based on the item in the document to which each word extracted by the morphological analysis unit 102 belongs. FIG. 3 shows an example of the item importance table 108. The item importance table 108 stores, for each item in the document, a weighting factor for correcting the importance of words appearing in that item. For example, in the item importance table 108, a weighting factor of "2.0" is assigned to "Title," which is considered to be an important item in the document, a weighting factor that is relatively larger than other items. On the other hand, in the item importance table 108, a weighting factor of "1.0" is assigned to "Footnote," which is considered to be a less important item in the document, a weighting factor that is smaller than other items. Note that the item importance table 108 is merely an example; various items can be defined and registered in the item importance table 108 as long as they reflect the importance intended by the document creator.

[0019] The importance correction unit 107 corrects the importance of a word calculated by the importance calculation unit 106 by multiplying the importance by a weighting factor corresponding to the item in which the word appears, which is registered in the item-specific importance table 108. In the case of a word that appears in multiple items in a document, the importance correction unit 107 obtains the largest weighting factor among the multiple items in which the word appears from the item-specific importance table 108 and multiplies the weighting factor by the importance of the word.

[0020] Returning to FIG. 1, the document vector generation unit 109 generates a document vector based on the importance of each word corrected by the importance correction unit 107 , and stores the document vector in the document vector storage unit 110 .

[0021] The search unit 111 performs document searches as document processing. In more detail, the search unit 111 includes a search statement acquisition unit 111a, a search processing unit 111b, and a search result output unit 111c.

[0022] The search statement acquisition unit 111a acquires a search statement that serves as a search condition for searching for documents. A search statement is a sentence that includes one or more search keywords. Note that a search statement may consist of only a single keyword. For example, the search statement acquisition unit 111a acquires a search statement input by a user via an input device, which will be described later. Note that the search statement acquisition unit 111a may also acquire a search statement from an external server, PC, or the like via a network.

[0023] The search processing unit 111b detects documents that match the content of the search statement acquired by the search statement acquisition unit 111a, using the document vector of each document stored in the document vector storage unit 110. Specifically, the search processing unit 111b detects documents that match the content of the search statement by calculating the similarity between the feature vector generated from the search statement and the document vector of each document stored in the document vector storage unit 110.

[0024] The search result output unit 111c outputs information indicating the results of the search performed by the search processing unit 111b.

[0025] Next, a description will be given of the hardware configuration of the document processing system 100. As shown in Fig. 4, the document processing system 100 includes, as its hardware configuration, a processor 120, a main memory unit 130, an auxiliary memory unit 140, a communication unit 150, an input unit 160, and an output unit 170. The main memory unit 130, the auxiliary memory unit 140, the communication unit 150, the input unit 160, and the output unit 170 are connected to the processor 120 via an internal bus 180.

[0026] The processor 120 includes a CPU (Central Processing Unit) and controls each unit of the document processing system 100. The processor 120 also executes programs stored in the auxiliary storage unit 140 to perform the processes of the document acquisition unit 101, the morphological analysis unit 102, the occurrence frequency calculation unit 104, the number of documents that appear calculation unit 105, the importance calculation unit 106, the importance correction unit 107, the document vector generation unit 109, and the search unit 111 of the document processing system 100.

[0027] The main memory unit 130 includes a RAM (Random Access Memory), and a program executed by the processor 120 is read from the auxiliary memory unit 140 to the main memory unit 130. This allows the main memory unit 130 to function as a work area for the processor 120. Note that the main memory unit 130 may read multiple programs from the auxiliary memory unit 140.

[0028] The auxiliary storage unit 140 includes a non-volatile memory such as an EEPROM (Electrically Erasable Programmable Read-Only Memory) or an HDD (Hard Disk Drive). The auxiliary storage unit 140 stores programs executed by the processor 120 and various data used in the processing of the processor 120. For example, the auxiliary storage unit 140 stores the above-mentioned word storage unit 103, the item importance table 108, and the document vector storage unit 110 as examples of the various data. In accordance with instructions from the processor 120, the auxiliary storage unit 140 supplies data used by the processor 120 to the processor 120 or the main storage unit 130 and stores the data supplied from the processor 120. The auxiliary storage unit 140 may store multiple programs for each function.

[0029] The communication unit 150 includes a NIC (Network Interface Card), and communicates with devices, databases, etc. connected to the network 10 in accordance with instructions from the processor 120.

[0030] The input unit 160 includes a keyboard, a mouse, a microphone, etc., and receives input operations from a user and outputs a signal corresponding to the received input operation to the processor 120. For example, the user operates the input unit 160 to input a search statement for searching for documents.

[0031] The output unit 170 includes a liquid crystal display, a speaker, and the like, and displays various information and outputs audio based on instructions from the processor 120. For example, the output unit 170 displays a screen showing the document search results.

[0032] Next, a detailed description will be given of the operation of the search processing executed by the document processing system 100. It is assumed that, prior to the search processing, data of a document set including multiple documents to be searched is stored in the document database 20. For example, when a user inputs a search statement from the input unit 160 and instructs the start of the search processing, the document processing system 100 starts the search processing shown in the flowchart of FIG.

[0033] When the search process starts, the document processing system 100 first executes a document vector generation process to generate a document vector for each document in the document set to be searched (step S100). The document vector generation process will be described in detail with reference to the flowchart in FIG.

[0034] First, the document acquisition unit 101 of the document processing system 100 acquires data of each document that constitutes a document set from the document database 20 (step S101).

[0035] Next, the morphological analysis unit 102 performs morphological analysis on the document to be processed among the documents constituting the document set, and extracts words contained in the document to be processed (step S102). Since morphological analysis can also identify the part of speech of a word, the morphological analysis unit 102 may exclude words of parts of speech such as particles and auxiliary verbs that do not represent the content of the document to be processed from the extraction targets, or may extract only noun words. Furthermore, the morphological analysis unit 102 also identifies the item within the document to be processed to which the extracted word belongs. In the following description, the document to be processed will also be referred to as the target document.

[0036] Next, the morphological analysis unit 102 determines whether format information is set for the target document (step S103). The format information here is text information such as a set phrase, header, and footer of the document, and is information that does not represent the content of the document.

[0037] If format information is not set in the target document (step S103; No), the process proceeds to step S106. On the other hand, if format information is set (step S103; Yes), the morphological analysis unit 102 performs morphological analysis on the text of the format information to extract words (step S104). Then, the morphological analysis unit 102 excludes the words extracted from the format information in step S104 from the words extracted from the target document in step S101 (step S105). Then, the process proceeds to step S106.

[0038] It should be noted that the document processing system 100 does not necessarily have to execute the processes of steps S103 to S105. After executing step S102, the document processing system 100 may skip steps S103 to S105 and execute step S106.

[0039] In step S106, the morphological analysis unit 102 assigns a unique ID within the document set to each word extracted from the target document, and registers each word in the word storage unit 103 in association with the word ID and the item in the target document in which the word appears. FIG. 7 is a diagram showing an example of a table recorded in the word storage unit 103 by the processing of step S106 when the target document is the above-mentioned document A (see FIG. 2). A table recorded in the word storage unit 103 is created for each document. FIG. 7 shows the table for document A, in which, for example, the word "cable" extracted from document A is registered together with the ID "001" assigned to the word and the items in which the word appears, "headline" and "body." Note that in this example, only noun words are extracted from document A.

[0040] 6, the above steps S102 to S106 are repeatedly executed until words are extracted from all documents constituting the document set acquired in step S101 (step S107; No). Then, when word extraction is completed for all documents (S107; Yes), the occurrence frequency calculation unit 104 refers to the word storage unit 103 and calculates the occurrence frequency in the target document of the document set, of each word extracted from the target document (step S108).

[0041] Next, the document appearance count calculation unit 105 refers to the word storage unit 103 and calculates, for each word extracted from the target document, the number of documents in which that word appears in the document set (step S109).

[0042] Next, the importance calculation unit 106 calculates the importance of each word extracted from the document to be processed by substituting the occurrence frequency calculated in step S108, the number of documents calculated in step S109, and the total number of documents constituting the document set into the above-mentioned formula (1) (step S110).The importance calculation unit 106 then records the calculated importance of each word in association with that word in the word storage unit 103.As a result, for the above-mentioned document A, the importance of each word is recorded in the table for document A in the word storage unit 103, as shown in FIG.

[0043] Returning to FIG. 6, next, the importance correction unit 107 refers to the item-by-item importance table 108 shown in FIG. 3, and corrects the importance calculated in step S110 for each word extracted from the target document based on the item in the processing document to which the word belongs (step S111).

[0044] Here, the processing of step S111 will be specifically described using the above-mentioned document A as an example. It is assumed that a table related to document A has been registered in the word storage unit 103 as shown in FIG. 7 by the processing up to step S110.

[0045] In this case, the item in which the word "Penetration" extracted from document A appears is "Summary," and it can be seen from the item importance table 108 shown in Fig. 3 that the weighting coefficient for the item "Summary" is "1.5." Therefore, the importance of the word "Penetration," which was "1," is corrected to "1.5" by multiplying it by this weighting coefficient, "1.5."

[0046] Furthermore, the word "transformer" extracted from document A appears in four items: "title," "summary," "headline," and "body." From the item importance table 108 shown in Figure 3, it can be seen that the weighting factor for the item "title" is the highest, at "2.0." Therefore, the importance of the word "transformer," which was originally "3," is corrected to "6.0" by multiplying it by the weighting factor "2.0."

[0047] Thereafter, the importance is corrected in the same manner for other words extracted from document A. Then, the importance of each word in document A is corrected as shown in FIG.

[0048] Returning to Fig. 6, the document vector generation unit 109 then generates a document vector representing the characteristics of the target document from the corrected importance of each word extracted from the target document stored in the word storage unit 103, and stores the document vector in the document vector storage unit 110 (step S112). This document vector is a vector whose components are the corrected importance of each word extracted from the target document. Note that the document vector may be generated using the TF-IDF method or a publicly available trained module such as Word2vec or Doc2Vec.

[0049] The above steps S108 to S112 are repeatedly executed until document vectors are generated for all documents constituting the document set (S113; No). Then, when document vectors are generated for all documents (S113; Yes), the document vector generation process ends.

[0050] Returning to FIG. 5, when the document vector generation process is completed, the search statement acquisition unit 111a acquires a search statement that serves as a search condition for searching documents (step S200).

[0051] Then, the search processing unit 111b generates a feature vector representing the features of the acquired search statement from the search statement (step S300). Specifically, the search processing unit 111b performs morphological analysis on the search statement to extract words, and generates the feature vector by vectorizing the set of extracted words. Note that the method by which the search processing unit 111b generates a feature vector from a search statement is not limited to this, and the feature vector may be generated by any known method.

[0052] Next, the search processing unit 111b calculates the similarity between the feature vector generated from the search query and each document vector stored in the document vector storage unit 110 (step S400). Specifically, this similarity is the cosine similarity, but the similarity may be calculated using other known measures.

[0053] Then, the search result output unit 111c outputs the search results based on the calculated similarity (step S500). For example, the search result output unit 111c displays a search result output screen on the liquid crystal display of the output unit 170, which shows a list of document contents and similarities in descending order of similarity, as shown in Fig. 9. Note that how the search results are output based on the calculated similarity is arbitrary, and for example, the search result output unit 111c may display only the document with the highest similarity as the search result. This completes the search process.

[0054] As described above, in the document processing system 100 according to the present embodiment, the importance correction unit 107 corrects the importance calculated from the frequency of occurrence of each word extracted from the target document according to the item in the target document to which each word belongs. Then, the document vector generation unit 109 generates a document vector for the target document based on the importance of each word after correction. This allows, for example, the importance of a word belonging to the title, which is an important item in the document, to be corrected to a large value even if the word's frequency of occurrence is low. Therefore, it is possible to generate a document vector that more closely reflects actual importance than conventional methods.

[0055] Furthermore, according to this embodiment, a weighting factor is set for each item in item importance table 108, and importance correction unit 107 corrects the importance by referencing item importance table 108 and assigning a weight according to the item to which each extracted word belongs. This makes it possible to easily correct the importance by simply setting the weighting factor.

[0056] Furthermore, according to this embodiment, for each extracted word that belongs to multiple items in the target document, the importance of that word is corrected by assigning the largest weight among the weights corresponding to the multiple items, thereby making it possible to correct the importance to an appropriate value.

[0057] (Variation) Although the embodiments of the present disclosure have been described above, modifications and applications in various forms are possible. It is up to the discretion of which parts of the configurations, functions, and operations described in the above embodiments to adopt. Furthermore, in addition to the above-described configurations, functions, and operations, further configurations, functions, and operations may be adopted. Furthermore, the configurations, functions, and operations described in the above embodiments can be freely combined.

[0058] In the above embodiment, the importance correction unit 107 corrects the importance by multiplying the calculated importance by a weighting coefficient corresponding to the item in which the word appears that is registered in the item-specific importance table 108, but the method of correcting the importance is not limited to this. For example, the importance correction unit 107 may correct the importance by adding to the importance a value corresponding to the item in the document in which the word appears.

[0059] In the above embodiment, the importance calculation unit 106 may calculate the importance of a word only from the frequency of occurrence of the word calculated by the occurrence frequency calculation unit 104. In this case, the document processing system 100 does not need to include the number-of-occurrence-documents calculation unit 105.

[0060] In the above embodiment, the document processing system 100 that performs search processing has been described, but the natural language processing that can be performed by the document processing system 100 is not limited to search processing. For example, the present disclosure is also applicable to a document processing system that generates document vectors and performs natural language processing other than search, such as document classification and machine summarization.

[0061] In the above embodiment, it is assumed that the documents acquired by the document acquisition unit 101 from the document database 20 are structured in advance and that the document processing system 100 can recognize the items in the documents. However, even in the case of unstructured documents whose items cannot be recognized, the document processing system 100 can generate document vectors in the same way by analyzing the text of the acquired documents and estimating the items in the documents. The estimation of the items in the documents here may be performed using a known method, and for example, the items may be estimated from the size of characters, decoration, position in the document, numbers at the beginning of sentences, etc.

[0062] The functions of the document processing system 100 described above can be realized by dedicated hardware or a conventional computer system. For example, a program executed by the processor 120 can be stored and distributed on a computer-readable, non-transitory recording medium, and the program can be installed on a computer to configure an apparatus that executes the above-described processes. Examples of such recording media include a flexible disk, a CD-ROM (Compact Disc Read-Only Memory), a DVD (Digital Versatile Disc), and an MO (Magneto-Optical Disc). The program may also be stored on a disk device of a server device on a communication network, such as the Internet, and downloaded to a computer, for example, by superimposing it on a carrier wave. Note that if the above-described functions are realized by an operating system (OS) or by cooperation between the OS and an application, only the parts other than the OS may be stored on a medium and distributed, or downloaded to a computer. The means for realizing the functions of the document processing system 100 are not limited to software; some or all of the functions may be realized by dedicated hardware including circuits.

[0063] In the above embodiment, the CPU of the processor 120 executes a program stored in the auxiliary storage unit 140, thereby functioning as each unit shown in FIG. 1 . However, in the present disclosure, the processor 120 may be dedicated hardware. Dedicated hardware may be, for example, a single circuit, a composite circuit, a programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. When the processor 120 is dedicated hardware, the functions of each unit may be implemented by individual hardware, or the functions of each unit may be implemented collectively by a single piece of hardware. Furthermore, some of the functions of each unit may be implemented by dedicated hardware, and the other functions may be implemented by software or firmware. In this way, the processor 120 can implement each of the above-described functions by hardware, software, firmware, or a combination thereof.

[0064] By applying an operating program that defines the operation of document processing system 100 according to the present disclosure to a computer such as an existing personal computer or information terminal device, it is possible to cause the computer to function as document processing system 100 according to the present disclosure. In addition, the method of distribution of such a program is arbitrary, and for example, the program may be stored on a computer-readable recording medium such as a CD-ROM (Compact Disk ROM), a DVD (Digital Versatile Disk), an MO (Magneto Optical Disk), or a memory card and distributed, or may be distributed via a communication network such as the Internet.

[0065] In the above-described embodiments, the relative positional relationships of the components or the conditions for implementation may be described, but these are merely examples in all respects and are not limited to those described in this specification. Therefore, countless variations not exemplified are contemplated within the scope of the technology disclosed in this specification. For example, the scope of the technology disclosed in this specification includes cases where at least one component is modified, added, or omitted.

[0066] Furthermore, unless a contradiction arises, a component described in the above embodiments as being provided with "one" may be provided with "one or more." Furthermore, each component is a conceptual unit, and includes cases where one component is made up of multiple structures, cases where one component corresponds to part of a structure, and even cases where multiple components are provided in one structure. Furthermore, each component includes structures with other structures or shapes as long as they perform the same function.

[0067] Furthermore, the descriptions in this specification are used for all purposes related to the present technology, and none of them are acknowledged as prior art. The present disclosure allows various embodiments and modifications without departing from the broad spirit and scope of the present disclosure. Furthermore, the above-described embodiments are intended to explain the present disclosure and do not limit the scope of the present disclosure. In other words, the scope of the present disclosure is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and the meaning of equivalent disclosures are deemed to be within the scope of the present disclosure.

[0068] Various aspects of the present disclosure are summarized below as appendices.

[0069] (Appendix 1) a word extraction unit that extracts words from a target document; an appearance frequency calculation unit that calculates the appearance frequency of each word extracted by the word extraction unit in the target document; an importance calculation unit that calculates the importance of each word extracted by the word extraction unit based on the calculated appearance frequency; an importance correction unit that corrects the importance calculated by the importance calculation unit based on the item in the target document to which each word extracted by the word extraction unit belongs; a document vector generation unit that generates a document vector of the target document based on the importance of each word corrected by the importance correction unit; A document processing system comprising: (Appendix 2) the importance correction unit corrects the importance by assigning a weight according to an item to which each of the words belongs; 10. A document processing system as set forth in claim 1. (Appendix 3) the importance correction unit corrects the importance of each of the words that belong to a plurality of items in the target document by assigning the largest weight among the weights corresponding to the plurality of items; 2. A document processing system as described in Appendix 2. (Appendix 4) the importance correction unit corrects the importance by referring to an item importance table in which a weighting factor to be assigned to a word belonging to each item is registered for each item; 2. A document processing system as described in Appendix 2. (Appendix 5) A word extraction unit extracts words from the target document, an occurrence frequency calculation unit calculates the occurrence frequency of each extracted word in the target document; an importance calculation unit calculates the importance of each extracted word based on the calculated appearance frequency; an importance correction unit corrects the calculated importance based on the item in the target document to which each extracted word belongs; a document vector generation unit that generates a document vector of the target document based on the importance of each corrected word; Document processing methods. (Appendix 6) Computer, a word extraction unit that extracts words from the target document; an appearance frequency calculation unit that calculates the appearance frequency of each word extracted by the word extraction unit in the target document; an importance calculation unit that calculates the importance of each word extracted by the word extraction unit based on the calculated appearance frequency; an importance correction unit that corrects the importance calculated by the importance calculation unit based on the item in the target document to which each word extracted by the word extraction unit belongs; a document vector generation unit that generates a document vector of the target document based on the importance of each word corrected by the importance correction unit; A program that functions as a [Explanation of symbols]

[0070] 10 network, 20 document database, 100 document processing system, 101 document acquisition unit, 102 morphological analysis unit, 103 word memory unit, 104 occurrence frequency calculation unit, 105 occurrence document number calculation unit, 106 importance calculation unit, 107 importance correction unit, 108 item-specific importance table, 109 document vector generation unit, 110 document vector memory unit, 111 search unit, 111a search condition acquisition unit, 111b search processing unit, 111c search result output unit, 120 processor, 130 main memory unit, 140 auxiliary memory unit, 150 communication unit, 160 input unit, 170 output unit, 180 internal bus

Claims

1. a word extraction unit that extracts words from a target document; an appearance frequency calculation unit that calculates the appearance frequency of each word extracted by the word extraction unit in the target document; an importance calculation unit that calculates the importance of each word extracted by the word extraction unit based on the calculated appearance frequency; an importance correction unit that corrects the importance calculated by the importance calculation unit based on the item in the target document to which each word extracted by the word extraction unit belongs; a document vector generation unit that generates a document vector of the target document based on the importance of each word corrected by the importance correction unit; A document processing system comprising:

2. the importance correction unit corrects the importance by assigning a weight according to an item to which each of the words belongs; The document processing system according to claim 1 .

3. the importance correction unit corrects the importance of each of the words that belong to a plurality of items in the target document by assigning the largest weight among the weights corresponding to the plurality of items; The document processing system according to claim 2 .

4. the importance correction unit corrects the importance by referring to an item importance table in which a weighting factor to be assigned to a word belonging to each item is registered for each item; The document processing system according to claim 2 .

5. A word extraction unit extracts words from the target document, an occurrence frequency calculation unit calculates the occurrence frequency of each extracted word in the target document; an importance calculation unit calculates the importance of each extracted word based on the calculated appearance frequency; an importance correction unit corrects the calculated importance based on the item in the target document to which each extracted word belongs; a document vector generation unit that generates a document vector of the target document based on the importance of each corrected word; Document processing methods.

6. Computer, a word extraction unit that extracts words from the target document; an appearance frequency calculation unit that calculates the appearance frequency of each word extracted by the word extraction unit in the target document; an importance calculation unit that calculates the importance of each word extracted by the word extraction unit based on the calculated appearance frequency; an importance correction unit that corrects the importance calculated by the importance calculation unit based on the item in the target document to which each word extracted by the word extraction unit belongs; a document vector generation unit that generates a document vector of the target document based on the importance of each word corrected by the importance correction unit; A program that functions as a

Images