Similarity calculation program and computer

Abstract

To clearly present a degree of similarity of text file pairs, to a user.SOLUTION: A program causes a computer to execute the steps of: extracting nouns from a plurality of text files and calculating appearances of the extracted nouns appearing in the text files; calculating, for each of the text files, TF-IDF of the nouns extracted from the text files, based on the appearances calculated for the text files; calculating inner product of feature vectors composed of the calculated TF-IDF, for each of text file pairs included in the text files; and displaying, on a display device, a similarity index based on the inner product calculated for at least some of the text file pairs.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a technique for generating and displaying information regarding the degree of similarity between multiple text files. [Background technology]

[0002] The document analysis device disclosed in Patent Document 1 vectorizes the content of a document, calculates a document vector, performs clustering based on the document vector, and selects a representative document that has the closest cosine distance to the cluster center point. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-166426 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there are cases where it is desirable to obtain desired information from a text file group containing a large number of text files, but it is not easy for a human to read all of the text files. Such text file groups include in-house technical information (technical documents) accumulated in chronological order, documents containing specific keywords extracted from a specified database (e.g., a patent database), etc. Therefore, it would be extremely useful to concisely present the similarity between text files included in the text file group using easily understandable rules.

[0005] The present disclosure has been made in consideration of the above points, and its purpose is to present the similarity between text files to the user in an easy-to-understand manner. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, a similarity calculation program according to a first embodiment of the present disclosure is characterized in that it causes a computer to execute the steps of extracting nouns from a plurality of text files and calculating the number of occurrences of each extracted noun in each text file; calculating the TF-IDF of the plurality of nouns extracted from the plurality of text files for each of the text files based on the number of occurrences calculated for each of the plurality of text files; calculating the inner product of a distinctiveness vector having the calculated TF-IDF as a component for a pair of text files included in the plurality of text files; and displaying a similarity index based on the inner product calculated for the pair of text files on a display device.

[0007] This allows the similarity between the pair of text files to be presented to the user in an easy-to-understand manner using a similarity index.

[0008] Furthermore, a second embodiment of the present disclosure is characterized in that, in the first embodiment, the similarity index is a value corresponding to the logarithm of a value based on the inner product.

[0009] This allows the similarity index displayed on the display device to be closer to the similarity perceived by humans, and therefore the similarity between the pair of text files can be presented to the user in a manner that is easier to understand.

[0010] Furthermore, a third embodiment of the present disclosure is a method according to the second embodiment, in which the similarity index is IS, the inner product is γ, D is a positive real number, m is a real number exceeding 1, and ε is 10^ (-4) When the following values ​​are used, the similarity index is characterized by being expressed by the following formula:

[0011] IS=D+log m (γ+ε) Moreover, the fourth embodiment of the present disclosure is characterized in that in the third embodiment, D=10 and m=10 in the formula.

[0012] As a result, if the similarity between a pair of text files is high, the similarity index will be close to 10, and if the similarity is low, the similarity index will be close to -20, allowing the similarity to be presented to the user concisely.

[0013] Furthermore, a fifth embodiment of the present disclosure is characterized in that, in the first embodiment, the computer further executes a step of displaying on the display device information identifying the text file pairs whose inner product is greater than a predetermined threshold value that is greater than 0.05 and less than 0.2.

[0014] Moreover, a sixth embodiment of the present disclosure is characterized in that, in the fifth embodiment, the predetermined threshold value is equal to or greater than 0.063 and equal to or less than 0.1.

[0015] Furthermore, a seventh embodiment of the present disclosure is characterized in that, in the first embodiment, nouns are extracted from the titles of the plurality of text files, an addition process is performed in which a predetermined seed value is added to the number of occurrences of the extracted nouns in the titles of each of the text files, and the TF-IDF is calculated based on the number of occurrences after the addition process.

[0016] As a result, since the title of a text file is likely to contain a noun that indicates an important characteristic of the text file, by adding a seed value to the number of occurrences of the noun extracted from the title, the important characteristics of the text file can be strongly reflected in the similarity index.

[0017] In addition, an eighth embodiment of the present disclosure is a computer comprising a memory unit that stores a similarity calculation program relating to any one of the first to seventh embodiments, and a control unit that executes the similarity calculation program stored in the memory unit. [Effects of the Invention]

[0018] According to the present disclosure, the similarity between pairs of text files can be presented to the user in an easy-to-understand manner. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a block diagram of a similarity calculation system including a computer that executes a similarity calculation program according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a flowchart illustrating the operation of a computer that executes a similarity calculation program according to an embodiment of the present disclosure. [Figure 3] FIG. 3 is a table illustrating the first list. [Figure 4] FIG. 4 is a table illustrating the nouns shown in the second list. [Figure 5] FIG. 5 is an explanatory diagram illustrating five text files for which TF-IDFs are calculated. [Figure 6] FIG. 6 is an explanatory diagram illustrating five text files for which characteristic degree vectors between the text files are calculated. [Figure 7] FIG. 7 is a table showing the TF-IDF, the feature vector, and the cosine similarity calculated for the text file shown in FIG. [Figure 8] FIG. 8 is a table showing similarity indices calculated for the text files shown in FIG. [Figure 9] FIG. 9 is an explanatory diagram illustrating a display on the display device. [Figure 10] FIG. 10 is a table showing several cosine similarities and their corresponding similarity indices. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. The following description of the preferred embodiments is merely exemplary in nature and is not intended to limit the present invention, its applications, or its uses.

[0021] 1 shows a similarity calculation system 1 that calculates a similarity index that indicates the mutual similarity between a plurality of text files (documents). The similarity calculation system 1 includes a computer 10 and a display device 20.

[0022] The computer 10 executes a similarity calculation program according to an embodiment of the present disclosure. The computer 10 includes a control unit 11 and a storage unit 12.

[0023] The control unit 11 calculates a similarity index indicating the mutual similarity between multiple text files by executing a similarity calculation program stored in the storage unit 12. The control unit 11 is also capable of communicating with an external network. The control unit 11 acquires multiple PDF files to be analyzed through communication with the external network and stores them in the storage unit 12.

[0024] The storage unit 12 stores various information such as input signals corresponding to user inputs, the similarity calculation program, various libraries received from an external network, PDF files to be analyzed, files and data generated by the control unit 11, etc. In this embodiment, Python is used to write the similarity calculation program, but other programming languages ​​may also be used. At least some of the functions of the storage unit 12 are realized by an SSD (Solid State Drive) that receives and stores input signals corresponding to user inputs.

[0025] The display device 20 displays the similarity index calculated by the control unit 11.

[0026] FIG. 2 shows the detailed operations performed by the computer 10 when the similarity calculation program is executed.

[0027] When the similarity calculation program is executed, a plurality of PDF files to be analyzed are acquired in advance and stored in the memory unit 12. Each PDF file stored in the memory unit 12 is given a title. This title is a natural language title such as "Analysis of Vibration" or "Effect of Oxygen Content." The PDF files are acquired through communication with an external network. The PDF files may also be acquired using a USB (Universal Serial Bus), an SD card reader, or an optical disc drive. The PDF files may also be generated based on user input.

[0028] <Step S101> In step S101, the control unit 11 extracts text from a plurality of PDF files to be analyzed that are stored in the storage unit 12, and creates a text file for each PDF file. The text file is created using the library pdfminer.

[0029] <Step S102> In step S102, the control unit 11 performs text shaping on each of the multiple text files created in step S101 to generate processed text files. The text shaping performed here is, for example, a process of deleting line breaks at the right end. The purpose of the text shaping performed here is to enable more accurate extraction of nouns, which will be described later.

[0030] <Step S103> In step S103, the control unit 11 extracts nouns from all the processed text files generated in step S102, i.e., the multiple text files to be analyzed, and calculates the number of times each extracted noun appears in each text file. The noun extraction here uses SudachiPy, developed by the Tokushima Works Artificial Intelligence NLP Laboratory. Note that libraries other than SudachiPy may also be used for noun extraction.

[0031] SudachiPy can cause the computer to extract nouns in three modes: a first mode in which compound words registered in a dictionary database are extracted without being divided as much as possible; a second mode in which compound words registered in the dictionary database are extracted by dividing them into shorter units; and a third mode in which at least some of the compound words registered in the dictionary database are extracted by dividing them into units shorter than those in the second mode. In this embodiment, the similarity calculation program includes a library of SudachiPy. The similarity calculation program is written to cause the computer to extract nouns in the first mode of SudachiPy.

[0032] SudachiPy allows a computer to select from three dictionary databases to reference. The similarity calculation program is written so that the computer references the largest dictionary database.

[0033] The control unit 11 generates a first list indicating extracted nouns and the number of times each noun appears for each text file, as illustrated in Fig. 3. In the example of Fig. 3, the nouns "details," "characteristics," "laser," "400," "distribution," "technology," "impact," and "creation" are extracted from a certain text file. In addition, the number of times "details" appears in this text file is calculated as 1, the number of times "characteristics" appears as 18, the number of times "laser" appears as 4, the number of times "400" appears as 2, the number of times "distribution" appears as 1, the number of times "technology" appears as 3, the number of times "impact" appears as 7, and the number of times "creation" appears as 1.

[0034] <Step S104> In step S104, the control unit 11 extracts nouns from the titles of all text files using the same method as in step S103. That is, the control unit 11 extracts nouns from the titles of all text files using SudachiPy in the first mode. If the extracted noun is included in the first list generated in step S103, the control unit 11 performs an addition process on the first list, in which a predetermined seed value is added to the number of occurrences of the extracted noun. If the extracted noun is not included in the first list generated in step S103, the control unit 11 adds the extracted noun to the first list and adds the seed value to the number of occurrences, which is 0 (the number of occurrences is the seed value). The seed value is set to, for example, 1 or 2. The title to which the seed value is added is the same as the title given to the PDF file before the text file was converted.

[0035] <Step S105> In step S105, the control unit 11 uses the first list of all text files after the addition process in step S104 to generate a second list showing the nouns extracted from all text files being analyzed and the number of times each noun appears in all text files. If a common noun is extracted in multiple text files, the total number of times it appears in the multiple text files is used as the number of times the noun appears in the second list. Figure 4 shows examples of nouns extracted from all text files, i.e., nouns shown in the second list.

[0036] <Step S106> In step S106, the control unit 11 calculates the TF-IDF of the multiple nouns extracted from all the text files for each text file based on the first list and the second list of all the text files. Specifically, the control unit 11 calculates the TF-IDF of each noun based on the number of occurrences of each noun calculated for each text file and the sum of the number of occurrences of each noun calculated for all the text files.

[0037] Here, the number of text files included in all text files is N, the number of each noun extracted from all text files is k, the number of text files in which the word with number k appears is d(k), and the word with number k is t. k Then, the IDF (Inverse Document Frequency) value is I D F is expressed by the following equation (1).

[0038]

number

[0039] For example, if all the text files are the five text files shown in Figure 5, then if the number k of "vibration" is 1 and the number k of "it" is 2, then N = 5, d(1) = 2, and d(2) = 5.

[0040] Therefore, the IDF value for "vibration" is as follows:

[0041]

number

[0042] Similarly, the IDF value for "it" is:

[0043]

number

[0044] Also, let the number of the text file be d, and let the noun t in the text file with number d be k The number of occurrences of f(t k , d), the noun t in the text file with number d k The TF value T indicates the occurrence rate of F is expressed by the following equation (2).

[0045]

number

[0046] In the example of FIG. 5, the occurrence counts of "vibration" and "sore" are as follows: f(vibration,1)=1 f(vibration,2)=0 f(vibration,3)=0 f(vibration,4)=2 f(vibration,5)=0 f(it,1)=1 f(it,2)=1 f(it,3)=2 f(it,4)=6 f(it,5)=1 Therefore, the denominator on the right side of the above equation (2) is calculated as 1+0+0+2+0+1+1+2+6+1=14.

[0047] The TF value indicating the occurrence rate of "vibration" in text file number 1 is 1 / 14, the TF value indicating the occurrence rate of "vibration" in text file number 4 is 2 / 14, and the TF value indicating the occurrence rate of "vibration" in text files number 2, number 3, and number 5 is 0 / 14.

[0048] Furthermore, the TF value indicating the occurrence rate of "that" in the text files numbered 1, 2, and 5 is 1 / 14, the TF value indicating the occurrence rate of "that" in the text file numbered 3 is 2 / 14, and the TF value indicating the occurrence rate of "that" in the text file numbered 4 is 6 / 14.

[0049] TF-IDF is the product of the IDF value and the TF value, i.e., T F * I D F is.

[0050] In the example of Figure 5, the TF-IDF of "vibration" in the text file numbered 1 is 0.693 / 14, the TF-IDF of "vibration" in the text file numbered 4 is 1.387 / 14, and the TF-IDF of "vibration" in the text files numbered 2, 3, and 5 is 0.

[0051] Also, the TF-IDF of "it" is 0 in all text files.

[0052] <Step S107> In step S107, the control unit 11 generates a characteristic score vector for each text file. The characteristic score vector is a vector whose components are the TF-IDFs of the nouns extracted from all the text files in step S106.

[0053] For example, if all the text files are the five text files shown in Figure 6, the TF-IDFs of "vibration," "it," and "oxygen" in each text file and the distinctiveness vectors of each text file will be as shown in Figure 7.

[0054] In Figure 7, the components of the distinctiveness vector of each text file are the TF-IDF of "vibration," the TF-IDF of "it," and the TF-IDF of "oxygen."

[0055] Specifically, the distinctiveness vector of text file number 1 is (0.707,0,0.707), the distinctiveness vector of text file number 2 is (0,0,0), the distinctiveness vector of text file number 3 is (0,0,0), the distinctiveness vector of text file number 4 is (1,0,0), and the distinctiveness vector of text file number 5 is (0.894,0,0.447).

[0056] <Step S108> In step S108, the control unit 11 calculates the inner product of the characteristic degree vectors calculated in step S107 for each text file pair included in all text files. Here, a text file pair means two (one pair) text files. The inner product of the characteristic degree vectors for each text file pair is the cosine similarity of each text file pair.

[0057] 7, the cosine similarity between text file number 1 and text file number 4 is 0.707, the cosine similarity between text file number 1 and text file number 5 is 0.949, and the cosine similarity between text file number 4 and text file number 5 is 0.894. The cosine similarities of other text file pairs are 0.

[0058] After step S107 is executed, and before step S108 is executed, it is determined whether the dimensions of the two feature degree vectors to be used for calculating the inner product match, and if they do not match, the dimensions may be unified by complementing the data.

[0059] <Step S109> In step S109, the control unit 11 calculates a similarity index based on the cosine similarity (the inner product of the characteristic vectors) for all text file pairs calculated in step S108. Here, if the cosine similarity calculated in step S108 is γ and the similarity index to be calculated is IS, the similarity index is expressed by the following formula (3). In this embodiment, D=10, m=10, and ε=10 -30 Let's say.

[0060] IS=D+log m (γ+ε) (3) When the cosine similarity is the value shown in Fig. 7, the similarity index is as shown in Fig. 8. In Fig. 8, the similarity index between text file No. 1 and text file No. 4 is 9.849, the similarity index between text file No. 1 and text file No. 5 is 9.977, and the similarity index between text file No. 4 and text file No. 5 is 9.951. The similarity index for other text file pairs is -20.

[0061] Furthermore, the control unit 11 causes the display device 20 to display, for each text file, the number of the text file whose similarity index with the text file exceeds a predetermined value, and the similarity index.

[0062] 9 illustrates an example of a display screen of the display device 20. In this example, the control unit 11 causes the display device 20 to display, in descending order of similarity index, the numbers of text files whose similarity index with the text file exceeds 8.8, and the similarity indexes of the text files. The control unit 11 also causes the display device 20 to display, for each text file, the number of text files whose similarity index with the text file exceeds 8.8. People tend to perceive text files whose similarity index is approximately 9 or greater as being similar to each other (related).

[0063] 9, Doc0132, Doc0076, ... Doc0002 are displayed as the numbers of text files whose similarity index with respect to the text file numbered Doc0001 exceeds 8.8. 9.83 is displayed as the similarity index between the text file numbered Doc0001 and the text file numbered Doc00, 9.12 is displayed as the similarity index between the text file numbered Doc0001 and the text file numbered Doc0076, and 8.93 is displayed as the similarity index between the text file numbered Doc0001 and the text file numbered Doc0002. Furthermore, 13 is displayed as the number of text files whose similarity index with respect to the text file numbered Doc0001 exceeds 8.8.

[0064] Similarly, Doc0420, Doc0001, and Doc0051 are displayed as the numbers of text files whose similarity index with respect to the text file numbered Doc0002 exceeds 8.8. 9.53 is displayed as the similarity index between the text file numbered Doc0002 and the text file numbered Doc0420, 8.93 is displayed as the similarity index between the text file numbered Doc0002 and the text file numbered Doc0001, and 8.83 is displayed as the similarity index between the text file numbered Doc0002 and the text file numbered Doc0051. Furthermore, 6 is displayed as the number of text files whose similarity index with respect to the text file numbered Doc0002 exceeds 8.8.

[0065] In FIG. 9, pairs of text files whose similarity index exceeds 8.8 are identified by the text file numbers. In this embodiment, the threshold value of the similarity index serving as the criterion for determining whether to display the text file on the display device 20 is set to 8.8, but other values ​​may be used. The threshold value of the similarity index serving as the criterion for determining whether to display the text file on the display device 20 is preferably a similarity index value corresponding to a cosine similarity of 0.05 to 0.2. That is, in step S109, it is preferable to display on the display device 20 information (numbers and similarity indexes) of pairs of text files whose cosine similarity (inner product of the characteristic vectors) is greater than a predetermined threshold value of 0.05 to 0.2. The range of the similarity index indicated by the above formula (3) corresponding to a cosine similarity of 0.05 to 0.2 is approximately 8.7 to 9.3. Therefore, it is preferable to set the threshold value of the similarity index to 8.7 to 9.3.

[0066] Furthermore, it is more preferable that the threshold value of the similarity index serving as the criterion for determining whether or not to display on the display device 20 is a similarity index value corresponding to a cosine similarity of 0.063 or more and 0.1 or less. That is, in step S109, it is more preferable that information (number and similarity index) of a text file pair whose cosine similarity (inner product of distinctiveness vectors) is greater than a predetermined threshold value of 0.063 or more and 0.1 or less is displayed on the display device 20. The range of the similarity index shown in the above formula (3) corresponding to a cosine similarity of 0.063 or more and 0.1 or less is approximately 8.8 or more and 9.0 or less, so it is more preferable that the threshold value of the similarity index is 8.8 or more and 9.0 or less.

[0067] If the similarity between the pair of text files is high, the similarity index will be close to 10, and if the similarity is low, the similarity index will be close to -20.

[0068] Figure 10 shows the relationship between cosine similarity and the similarity index. When the cosine similarity is 0.1 or higher, humans tend to perceive the two objects as similar. When the cosine similarity is 0.10, the similarity index is 9.

[0069] In this way, since the similarity index, which is closer to the similarity perceived by humans than the cosine similarity, is displayed on the display device 20, the user can more easily recognize the similarity between the text file pair than when the cosine similarity is displayed.

[0070] Therefore, according to this embodiment, the similarity between a pair of text files can be presented to the user in an easy-to-understand manner using a similarity index.

[0071] Furthermore, according to this embodiment, when past internal company documents are the subject of analysis, it becomes easier to extract necessary information from the past internal company documents or information that is difficult to notice through human memory. This makes it easier to use the extracted information as a trigger for new technological development. It also facilitates technology transfer and contributes to preventing technology transfer from becoming dependent on a specific individual. Furthermore, patent documents may also be the subject of analysis in this embodiment. This makes it easier to organize patent documents.

[0072] Furthermore, since the title of a text file is likely to contain a noun that indicates an important feature of the text file, in step S104, by adding a seed value to the number of occurrences of the noun extracted from the title, the important features of the text file can be strongly reflected in the similarity index.

[0073] In this embodiment, D=10, m=10, and ε=10 -30 The value IS expressed by the above formula (3) is used as the similarity index. However, when D, m, and ε are set to other values, the value IS expressed by the above formula (3) may also be used as the similarity index. ε is provided to prevent the antilogarithm of the logarithmic part from becoming 0, and is set to 10^ (-4) It is preferable to set it to the following value:

[0074] Although the similarity index is the value IS expressed by the above formula (3), other values ​​may be used as long as they correspond to the logarithm of the value based on the cosine similarity. For example, the value expressed by the following formula may be used. In this case, D=10, m=10, ε=10 -30 Alternatively, D, m, and ε may be set to other values.

[0075]

number

[0076] In the above embodiment, in step S109, the control unit 11 displays on the display device 20 only the similarity indices calculated for some of all text file pairs, specifically, those exceeding 8.8. However, when the number of text files to be analyzed is small, the control unit 11 may display on the display device 20 only the similarity indices calculated for all text file pairs. Furthermore, the control unit 11 may display on the display device 20 information (numbers and similarity indices) of text file pairs of all text file pairs, specifically, those having similarity indices that exceed a predetermined threshold value less than 8.8. On the other hand, if the control unit 11 displays on the display device 20 information (numbers and similarity indices) of text file pairs of all text file pairs whose similarity indices exceed 8.8, the amount of text file information displayed may be too large. The predetermined threshold may be set to a value greater than 8.8, such as 9.0. In other words, the control unit 11 may display on the display device 20 only the information (numbers and similarity indices) of text file pairs whose similarity indices exceed a predetermined threshold value greater than 8.8.

[0077] In the above embodiment, the present invention is applied to a text file created based on a PDF file, but the present invention can also be applied to text files created based on files of other formats, such as Word files, HTML files, etc. When creating a text file from an HTML file, the pdfminer library is not required. [Industrial Applicability]

[0078] The present disclosure is useful as a technique for generating and displaying information regarding the degree of similarity between multiple text files. [Explanation of symbols]

[0079] 10. Computers 11 Control section 12 Storage section 20 Display device

Claims

1. extracting nouns from a plurality of text files and calculating the number of occurrences of each extracted noun in each text file; calculating, for each of the plurality of text files, a TF-IDF of a plurality of nouns extracted from the plurality of text files based on the number of occurrences calculated for each of the plurality of text files; calculating an inner product of characteristic vectors having the calculated TF-IDF as components for each pair of text files included in the plurality of text files; and displaying, on a display device, a similarity index based on the inner product calculated for the pair of text files.

2. 2. The similarity calculation program according to claim 1, The similarity calculation program is characterized in that the similarity index is a value corresponding to the logarithm of a value based on the inner product.

3. 3. The similarity calculation program according to claim 2, The similarity index is IS, the inner product is γ, D is a positive real number, m is a real number greater than 1, and ε is 10^ （－４） A similarity calculation program characterized in that, when the following values ​​are used, the similarity index is expressed by the following formula: IS = D + log ｍ (c+e)

4. 4. The similarity calculation program according to claim 3, In the above formula, A similarity calculation program characterized in that D=10 and m=10.

5. 2. The similarity calculation program according to claim 1, The similarity calculation program further causes the computer to execute a step of displaying, on the display device, information specifying the text file pairs whose inner product is greater than a predetermined threshold value of 0.05 or more and 0.2 or less.

6. 6. The similarity calculation program according to claim 5, The similarity calculation program is characterized in that the predetermined threshold is equal to or greater than 0.063 and equal to or less than 0.

1.

7. 2. The similarity calculation program according to claim 1, extracting nouns from the titles of the plurality of text files; performing an addition process of adding a predetermined seed value to the number of occurrences of the extracted nouns in the titles of each of the text files; The similarity calculation program is characterized in that the TF-IDF is calculated based on the number of occurrences after the addition process.

8. a storage unit that stores the similarity calculation program according to any one of claims 1 to 7; a control unit that executes the similarity calculation program stored in the storage unit.

Images