Document data processing device and program

Abstract

To provide a document data processing device and program, which improve extraction accuracy of document data with similar content.SOLUTION: A document data processing device is provided, comprising a document data acquisition unit 21 for acquiring document data, and an error code assigning unit 22 configured to newly assign the error code to document data not including the error code but including a co-occurrence word on the basis of the co-occurrence word of the error code included in document data including the error code.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a document data processing apparatus and a program for adding a specific keyword such as an error code to document data. [Background technology]

[0002] Conventionally, when a problem occurs with a device such as a printing device, the user creates a problem report and sends it to the device manufacturer, or sends text data via an inquiry form on the manufacturer's website.

[0003] The manufacturer stores document data such as defect reports sent by users on a server or the like and analyzes it. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-287020 Summary of the Invention [Problem to be solved by the invention]

[0005] When analyzing the above-mentioned defect reports, etc., it may be necessary to extract document data containing similar content from document data stored on a server. When extracting such similar document data, a search is performed using specific keywords, such as an error code displayed on a device screen when a defect occurs, and the degree of matching of these specific keywords can have a significant impact on the extraction accuracy.

[0006] However, for example, if an error code is displayed on the device screen when a malfunction occurs, but the error code is not written in the malfunction report, or if the error code is not written correctly due to a typo, it becomes difficult to extract the document data as document data similar to document data written about the same malfunction.

[0007] Patent Document 1 proposes a method for eliminating missed searches of document data by expanding words extracted from document data into synonyms that have similar meanings and are appropriate for the context.

[0008] However, when searching using a unique word such as an error code that cannot be replaced, the method described in Patent Document 1 cannot be applied.

[0009] An object of the present invention is to provide a document data processing device and a program that can improve the accuracy of extracting document data with similar contents. [Means for solving the problem]

[0010] The document data processing device of the present invention includes a document data acquisition unit that acquires document data, and a specific keyword assignment unit that assigns a new specific keyword to document data that does not contain the specific keyword but does contain co-occurring words, based on co-occurring words of the specific keyword contained in document data that includes the specific keyword.

[0011] The document data processing program of the present invention causes a computer to execute the steps of acquiring document data and assigning a new specific keyword to document data that does not contain the specific keyword but does contain co-occurring words, based on co-occurring words of the specific keyword contained in document data that includes the specific keyword. [Effects of the Invention]

[0012] According to the document data processing device and program of the present invention, a specific keyword is newly assigned to document data that does not contain the specific keyword but does contain co-occurring words, based on co-occurring words of the specific keyword contained in document data that contains the specific keyword, thereby improving the accuracy of extracting document data with similar content. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram showing a schematic configuration of a document search system using a first embodiment of the document data processing device of the present invention. [Figure 2] 1 is a flowchart illustrating a method for assigning an error code in a document search system according to a first embodiment of the present invention; [Figure 3] FIG. 10 is a diagram showing an example of a table A that associates document IDs of merged document data with error codes included in the merged document data. [Figure 4] An example of Table B, which summarizes the TF-IDF calculation results for each word. [Figure 5] A diagram showing an example of table C that associates the top X words by TF-IDF value with the document IDs of the combined document data. [Figure 6] A diagram showing an example of Table D, which associates error codes with the words that correspond to those error codes. [Figure 7] 10 is a flowchart illustrating a method for assigning an error code in a document search system according to a second embodiment of the present invention. [Figure 8] 10 is a flowchart illustrating a method for assigning an error code in a document search system according to a second embodiment of the present invention. [Figure 9] An example of Table E, which lists the error codes and the co-occurrence strength of all words. [Figure 10] An example of Table F, which summarizes the weight of each word taking into account the sentence length of the document data. [Figure 11] A diagram for explaining a method for calculating the average and standard deviation of the sum of the weighted co-occurrence intensities of each word corresponding to each error code. [Figure 12]FIG. 10 is a diagram showing an example of a table H showing threshold conditions corresponding to each error code. [Figure 13] An example of Table I, which shows the sum of the weighted co-occurrence strengths of each word for each error code for each document data. [Figure 14] FIG. 10 is a diagram illustrating an example of a method for determining whether an error code is assigned using a threshold condition corresponding to the error code. DETAILED DESCRIPTION OF THE INVENTION

[0014] A document retrieval system using a first embodiment of a document data processing device of the present invention will be described in detail below with reference to the drawings. Fig. 1 is a block diagram showing a schematic configuration of a document retrieval system 1 of this embodiment.

[0015] As shown in FIG. 1, the document retrieval system 1 of this embodiment includes a document database 10, a document data processing device 20, a document data input device 30, and a document data retrieval device 40.

[0016] The document database 10 and the document data processing device 20, the document database 10 and the document data input device 30, and the document database 10 and the document data search device 40 are connected by wired or wireless communication lines, and are configured to be able to communicate with each other. The communication lines may be a LAN (Local Area Network) or an Internet line.

[0017] The document database 10 stores a plurality of document data. In this embodiment, the document database 10 stores, as document data, defect reports for printing devices such as inkjet printers and text data sent from inquiry forms on the websites of printing device manufacturers.

[0018] The above-mentioned defect reports and text data may or may not include an error code for the printing device. The document database 10 of this embodiment stores both document data that includes an error code and document data that does not include an error code. Note that the document data is not limited to the above-mentioned defect reports and text data, and may be any document data that may or may not include a predetermined specific keyword, such as an error code.

[0019] Specifically, the document database 10 includes a storage medium such as a semiconductor memory and a hard disk for storing document data, a CPU (Central Processing Unit) that controls the reading and writing of document data from the storage medium, and a communication IF (Interface).

[0020] The document data processing device 20 includes a document data acquisition section 21 and an error code assignment section 22. In this embodiment, the error code assignment section 22 corresponds to the specific keyword assignment section of the present invention.

[0021] The document data acquisition unit 21 acquires the document data read out from the document database 10 .

[0022] The error code assigning unit 22 assigns a new error code to document data that does not contain an error code but does contain the co-occurring word, based on the co-occurring word of the error code contained in the document data that includes the error code among the document data acquired by the document data acquisition unit 21.

[0023] Co-occurring words of an error code are words that are often used together with an error code, and are obtained by the error code assigning unit 22.

[0024] The error code assigning unit 22 of this embodiment counts the number of co-occurring words included in document data that does not contain an error code, and assigns a new error code to the document data that does not contain the error code based on the counted number of co-occurring words. Note that a specific method of assigning an error code in this embodiment will be described in detail later.

[0025] Document data processing device 20 is configured as a computer, and includes storage media such as semiconductor memory and a hard disk, a CPU, and a communications interface. In document data processing device 20 of this embodiment, the functions of document data acquisition unit 21 and error code assignment unit 22 are realized by the CPU executing a document data processing program pre-stored in the storage media.

[0026] In this embodiment, the functions of the document data acquisition unit 21 and the error code assignment unit 22 are realized by a document data processing program stored in advance as described above, but this is not limited to this. For example, some or all of the functions may be configured from hardware such as an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or other electrical circuits.

[0027] Document data input device 30 accepts input of document data and stores the document data in document database 10. Document data input device 30 may be, for example, a terminal device of a user who uses a printing device, or a printing device capable of outputting defect reports, or may be a terminal device of an administrator who collects and manages defect reports and text data sent from inquiry forms.

[0028] Document data input device 30 is configured as a computer, and includes storage media such as semiconductor memory and a hard disk, a CPU, and a communication IF.

[0029] The document data search device 40 searches for document data stored in the document database 10. For example, the document data search device 40 searches for and extracts document data that includes a predetermined error code from the document data stored in the document database 10. This document data that includes an error code can also include document data to which an error code has been assigned by the error code assignment unit 22 of the document data processing device 20 described above.

[0030] The document data search device 40 is configured as a computer, and includes a storage medium such as a semiconductor memory and a hard disk, a CPU, and a communication IF.

[0031] Next, an error code assignment method in the document data processor 20 of the first embodiment will be described with reference to the flowchart shown in FIG.

[0032] First, as a prerequisite, the document database 10 stores a sufficient number of both document data including error codes and document data not including error codes to enable the error code assignment method described later to be implemented.

[0033] Each document data is composed of a document ID for identifying the document data and body data.

[0034] First, the document data acquisition unit 21 extracts and acquires all document data that contain an error code in the body data from among the many document data stored in the document database 10 (S10). At this time, an arbitrary period may be specified, and document data stored during that period may be extracted.

[0035] Next, the error code assigning unit 22 identifies document data whose body data contains the same error code from among the document data containing error codes acquired by the document data acquiring unit 21, and generates combined document data by combining the body data of the identified document data (S12). A new document ID is assigned to the generated combined document data. Note that if the body data of the document data contains multiple error codes, document data whose all error codes match is identified, and the body data of the identified document data is combined to generate combined document data.

[0036] Then, the error code assigning unit 22 creates a table A (S14) that associates the document IDs of the merged document data with the error codes included in the merged document data, as shown in Fig. 3. "X-0000" and "X-0001" shown in Fig. 3 are document IDs, and "a1-0000-0000" and "d99-0123-4567" are error codes. The document ID "X-0000" includes two error codes.

[0037] Next, the document data acquisition unit 21 extracts and acquires document data that does not include an error code in the body data from among the many document data stored in the document database 10 (S16).

[0038] The error code assigning unit 22 then performs preprocessing on the group of document data that does not contain error codes extracted by the document data acquiring unit 21 and the group of combined document data described above, and then calculates the TF-IDF of the words contained in each document data that does not contain error codes, as well as the TF-IDF of the words contained in each combined document data. The error code assigning unit 22 then creates a table B that summarizes the TF-IDF calculation results, as shown in Fig. 4 (S18). "Engine," "panel," and "multiple feed," as shown in Fig. 4, are words contained in the document data and combined document data that contain error codes.

[0039] Here, the above-mentioned preprocessing refers to processing for extracting appropriate words from document data and combined document data, and includes, for example, processing for deleting unnecessary words, processing for standardizing words with variations in spelling, and processing for dividing documents in the main text data into words.

[0040] TF-IDF is a method for evaluating the importance of words contained in document data. TF-IDF can be calculated, for example, using the following formula:

number

number

number

[0041] It should be noted that the total number of documents here refers to all document data acquired by the document data acquisition unit 21.

[0042] Furthermore, in this embodiment, the TF-IDF is calculated, but the present invention is not limited to this, and any index may be used as long as it is an index for evaluating the importance of words included in document data.

[0043] Next, the error code assigning unit 22 extracts the top X words with TF-IDF values ​​from among the words included in each merged document data in table B shown in Fig. 4, and creates table C as shown in Fig. 5, which associates the extracted words (word 1, word 2, etc.) with the document IDs of the merged document data (S20). Note that X can be set to any value equal to or greater than 1, but a value that ensures extraction accuracy is set.

[0044] Thereafter, the error code assigning unit 22 associates the document ID of the merged document data shown in Table C shown in Fig. 5 with the merged document ID shown in Table A shown in Fig. 3 to merge Table C and Table A, and creates Table D, which associates error codes with words corresponding to those error codes, as shown in Fig. 6 (S22). Note that the words corresponding to each error code shown in Table D are co-occurring words corresponding to that error code.

[0045] Then, for each piece of document data that does not contain an error code acquired by the document data acquisition unit 21, the error code assignment unit 22 counts the co-occurring words corresponding to each error code in Table D shown in Figure 6 and determines whether it contains Y or more (S24).If predetermined document data that does not contain an error code contains Y or more co-occurring words corresponding to a specific error code (S24, YES), the error code assignment unit 22 assigns the specific error code to the end of the text data of the predetermined document data (S26).Then, the document data to which the error code has been assigned is stored again in the document database 10 under the same document ID (S28).Note that Y can be set to any value greater than or equal to 1, but a value that ensures extraction accuracy is set.

[0046] On the other hand, in S22, if there is no predetermined document data that contains Y or more co-occurring words corresponding to a specific error code but does not contain an error code (S24, NO), the process ends.

[0047] When document data is searched for by the document data search device 40, the document data to which the error code re-stored as described above is added is searched for.

[0048] According to the document search system 1 of this embodiment, a new error code is assigned to document data that does not contain the error code but does contain the co-occurring word, based on the co-occurring word of the error code contained in the document data that contains the error code, thereby improving the accuracy of extracting document data with similar content.

[0049] In addition, in this embodiment, error codes are assigned based on the number of co-occurring words of the error code contained in document data that does not contain an error code, so new error codes can be assigned by a simple process that simply counts the number of co-occurring words.

[0050] Next, a document search system using a second embodiment of the document data processing device of the present invention will be described in detail. The schematic configuration of the document search system of the second embodiment is similar to that of the document search system of the first embodiment shown in Figure 1. However, the document search system of the second embodiment differs from the document search system of the first embodiment in the method of assigning error codes in the document data processing device 20.

[0051] In the first embodiment, as described above, if document data that does not contain an error code contains Y or more co-occurring words corresponding to a specific error code, the document data is assigned the specific error code.However, with this error code assignment method, the longer the document data is, i.e., the more words it contains, the higher the probability that it will contain co-occurring words corresponding to an error code, and the more likely it is that an error code will be assigned.

[0052] Therefore, there is a risk that the error code will be assigned to document data that is not actually related to the error code, which may lead to a decrease in the accuracy of assigning the error code.

[0053] Therefore, in the second embodiment, the co-occurrence strength of each word (each co-occurring word) corresponding to the error code and the weight of each word taking into account the sentence length of the document data are calculated and used to compensate for the shortcomings of the error code assignment method of the first embodiment described above. Note that the co-occurrence strength is a value that represents the rate at which, for example, two words, "A" and "B," appear together in one document data. The higher the co-occurrence strength, the higher the rate at which one word is used in document data in which the other word is also used. In this embodiment, the above-mentioned word "A" corresponds to the error code, and the word "B" corresponds to a word contained in the document data.

[0054] The error code assignment method of the second embodiment will be described below with reference to the flowcharts shown in FIGS.

[0055] First, the document data acquisition unit 21 reads and acquires all document data stored in the document database 10 (S30).

[0056] Next, the error code assigning unit 22 calculates the co-occurrence strength of all words for each error code by calculating, for example, the Jaccard coefficient for the words included in all the document data acquired by the document data acquiring unit 21 (S32).

[0057] Then, the error code assigning unit 22 determines Z words with the highest co-occurrence strength for each error code as co-occurrence words corresponding to each error code (S34). At this time, the error code assigning unit 22 sets the co-occurrence strength of words that are not determined as co-occurrence words to zero, and creates a table E that summarizes the co-occurrence strength of each error code and all words, as shown in Fig. 9 (S36). The words corresponding to each error code in table E (x motor, y sensor, ... jam, etc.) are all words contained in all document data.

[0058] In Table E shown in FIG. 000_xモータ" indicates the co-occurrence strength, and the subscript "000" indicates the co-occurrence strength "a 000_xモータ " represents the numerical value of the error code corresponding to "x motor" and the co-occurrence strength "a 000_xモータ ". Also, for example, "a 001_ジャム " has a value of zero because the word "jam" was not determined as a co-occurrence word with the error code "A-001" or did not co-occur even once in all the document data. In this embodiment, the co-occurrence strength of words that were not determined as co-occurrence words is set to zero, which reduces the number of subsequent calculation processes.

[0059] Furthermore, Z can be set to any value equal to or greater than 1, but a value that can ensure extraction precision is set.

[0060] The Jaccard coefficient mentioned above is a known index, but we will briefly explain how to calculate it. For example, suppose that the error code "A-000" is contained in 80 pieces of document data, and the word "x motor" is contained in 110 pieces of document data. Furthermore, suppose that both the error code "A-000" and the word "x motor" are contained in 20 pieces of document data. In this case, if the set of document data containing the error code "A-000" is "A" and the set of document data containing "x motor" is "B", the Jaccard coefficient is calculated as follows, and is 0.125. A∩B / A∪B=(A∩B) / {(A+B)-(A∩B)} =(20 / 80+100-20)=0.125

[0061] In this embodiment, the Jaccard coefficient is calculated as the co-occurrence strength, but the present invention is not limited to this, and other known indices may be calculated.

[0062] Next, the error code assigning unit 22 calculates the weight of each word contained in all the document data, taking into consideration the sentence length of the document data, and creates a table F as shown in FIG. 10 (S38).

[0063] In Table F shown in FIG. 000_xモータ " indicates a weight taking into account the sentence length, and the meaning of the subscripts is the same as the co-occurrence strength described above.

[0064] Here, the weight of each word in the document data is a value that represents how important the word contained in the document data is in expressing the document data. Furthermore, by calculating the weight of each word taking into account the length of the document data, the longer the document data, the lower the importance of the word, and vice versa.

[0065] For example, Okapi BM25 can be used to weight each word taking into account the sentence length of the document data. Okapi BM25 can be calculated based on the following formula. The score obtained from the formula below is the importance of each word in each document data.

number

number

number

[0066] In this embodiment, Okapi BM25 is calculated as the weight of each word taking into account the sentence length of the document data. However, this is not limiting, and the above-mentioned TF-IDF or other well-known indices may also be calculated.

[0067] Next, the error code assigning unit 22 identifies document data containing error codes from all document data, and groups the document data containing error codes by error code (S40). Then, a table of the weight of each word in the document data grouped for a specific error code is created (S42). Figure 11A shows, for example, table G-1 of the weight of each word in document data grouped for error code "A-000".

[0068] Then, the error code assigning unit 22 extracts the co-occurrence strength of each word corresponding to a specific error code from table E shown in FIG. 9, and multiplies the co-occurrence strength of each word corresponding to that error code by the weight of each word in the document data grouped by that error code (S44).

[0069] Specifically, the error code assigning unit 22 extracts the row with error code "A-000" from table E shown in Fig. 9. Then, as shown in Fig. 11B, the error code assigning unit 22 multiplies table G-2 of the co-occurrence strength of each word in the extracted row by table G-1 of the weight of each word in the document data with error code "A-000" created in S42, to create table G-3 shown in Fig. 11C. The value obtained by multiplying the co-occurrence strength of each word by the weight is called weighted co-occurrence strength.

[0070] Thereafter, the error code assigning unit 22 calculates the sum of the weighted co-occurrence intensities of each word for each document data containing the error code "A-000" (S46), as shown in Table G-3. Note that "i" in the sum of the weighted co-occurrence intensities shown in Table G-3 represents each word included in the document data.

[0071] Then, the error code assigning unit 22 calculates the average value μ and standard deviation σ of the sum of the weighted co-occurrence intensities for the error code “A-000” (S48).

[0072] Then, the error code assigning unit 22 performs the processes of S40 to S48 for each error code in the same manner to calculate the average value μ and standard deviation σ of the sum of the weighted co-occurrence intensities corresponding to each error code.

[0073] Next, the error code assignment unit 22 calculates the threshold condition μ-Nσ for determining whether or not to assign each error code to the document data using the average value μ and standard deviation σ of the sum of the weighted co-occurrence intensities corresponding to each error code calculated in S48, and creates a table H as shown in Figure 12 (S50).

[0074] Next, the error code assigning unit 22 uses Table E shown in Fig. 9 and Table F shown in Fig. 10 to calculate the sum of the weighted co-occurrence intensities of each word for each error code for each document data, as shown in Fig. 13, to create Table I (S52). Note that "i" in the sum of the weighted co-occurrence intensities shown in Table I represents each word included in the document data, and "j" represents the error code.

[0075] Then, the error code assigning unit 22 compares the sum of the weighted co-occurrence intensities of each word for each error code of each document data included in Table I with the threshold conditions for each error code included in Table H shown in Figure 12, and checks whether the sum of the weighted co-occurrence intensities satisfies the threshold conditions for each error code (S54).

[0076] If there is a sum of weighted co-occurrence intensities that satisfies the threshold condition for each error code (S54, YES), the error code assigning unit 22 checks whether or not an error code is included in the document data having that sum of weighted co-occurrence intensities (S56).If there is no error code included in the document data (S56, YES), the error code assigning unit 22 assigns a new error code to the document data that corresponds to the sum of weighted co-occurrence intensities that satisfies the threshold condition (S58).

[0077] Specifically, as shown in FIG. 14, for example, the error code assigning unit 22 determines whether the sum of the weighted co-occurrence intensities of the words for the error code "A-111" of the document data "X-001" satisfies the threshold condition "μ A-111 -Nσ A-111 If the error code is equal to or greater than ", it is determined that the threshold condition is met. If the document data "X-001" does not contain an error code, a new error code "A-111" is added to the end of the text data of the document data "X-001", as shown in FIG. 14.

[0078] Then, the document data to which the error code has been assigned is stored again in the document database 10 with the same document ID (S60).

[0079] On the other hand, if there is no document data that satisfies the threshold conditions for each error code in S54, or if it is determined in S58 that the document data contains an error code, the process ends without assigning an error code.

[0080] When document data is searched for by the document data search device 40, the document data to which the error code re-stored as described above is added is searched for.

[0081] According to the document search system of the second embodiment, as described above, the co-occurrence strength of each word (each co-occurring word) corresponding to the error code and the weight of each word taking into account the sentence length of the document data are calculated and used, thereby further improving the accuracy of extracting document data containing the same error code.

[0082] It should be noted that the above first and second embodiments are examples in which a specific keyword of the present invention is used as an error code, but the specific keyword of the present invention is not limited to an error code and may be any other predetermined specific keyword.

[0083] The present invention further discloses the following supplementary notes. (Addendum)

[0084] In the document data processing device of the present invention, the specific keyword assignment unit counts the number of co-occurring words contained in document data that does not contain a specific keyword, and based on the count value of the number of co-occurring words, can assign a new specific keyword to document data that does not contain a specific keyword.

[0085] In the document data processing device of the present invention, the specific keyword assignment unit can assign a new specific keyword to document data that does not contain the specific keyword based on the co-occurrence strength of the words contained in all document data acquired by the document data acquisition unit with the specific keyword and the weight of the words contained in all document data based on the sentence length of each document data.

[0086] In the document data processing device of the present invention, the specific keyword can be an error code. [Explanation of symbols]

[0087] 1 Document Search System 10 Document Database 20 Document data processing device 21 Document data acquisition unit 22 Error code assignment section 30 Document data input device 40 Document data search device

Claims

1. A document data processing device that processes document data containing keywords in sentences, the document data being searched based on the keywords, comprising: a document data acquisition unit that acquires the document data; an extraction unit that extracts co-occurring words of the specific keyword from the document data including the specific keyword acquired by the document data acquisition unit; and a specific keyword assigning unit that adds the specific keyword to document data that does not contain the specific keyword but does contain the co-occurring word based on the co-occurring word extracted by the extracting unit.

2. A document data acquisition unit that acquires document data; a specific keyword assigning unit that assigns the specific keyword to document data that does not include the specific keyword but does include the co-occurring word, based on a co-occurring word of the specific keyword included in document data that includes the specific keyword; The specific keyword assignment unit counts the number of co-occurring words included in document data that does not contain the specific keyword, and assigns the specific keyword to document data that does not contain the specific keyword based on the count value of the number of co-occurring words.

3. A document data acquisition unit that acquires document data; a specific keyword assigning unit that assigns the specific keyword to document data that does not include the specific keyword but does include the co-occurring word, based on a co-occurring word of the specific keyword included in document data that includes the specific keyword; The specific keyword assignment unit assigns the specific keyword to document data that does not contain the specific keyword based on the co-occurrence strength of the words contained in all of the document data acquired by the document data acquisition unit with the specific keyword and weights of the words contained in all of the document data based on the sentence length of each document data.

4. A document data processing program for processing document data containing keywords in sentences, the document data being searched based on the keywords, comprising: acquiring the document data; extracting co-occurring words of the specific keyword from document data containing the acquired specific keyword; and a step of newly including the specific keyword in document data that does not include the specific keyword but does include the co-occurring word, based on the extracted co-occurring word.

Images