Document processing device

Abstract

Provided is a document processing device including a metadata assignment unit, a clustering unit, a difference extraction unit, and an output unit. The metadata assignment unit converts each of a plurality of documents into a vector on the basis of word occurrence counts in the document, and assigns, as metadata to the corresponding document, the vector and an issue time of the document. The clustering unit clusters the documents on the basis of similarity in the word occurrence counts represented by the vectors assigned by the metadata assignment unit. For each cluster formed by the clustering unit, the difference extraction unit rearranges the documents in order of issue time and extracts differences between the documents rearranged in that order. The output unit outputs the differences between the documents.

Description

Document processing device

[0001] The embodiment relates to a document processing device.

[0002] A well-known document is a document that describes content to be made known to a user. The well-known document may include, for example, an update notice document that notifies the user to perform a software update. With the increase in cyberattacks in recent years, the importance of software update operations related to information security has increased, and the importance of well-known documents that notify users of such software update operations has also increased.

[0003] Hiroki Nakanishi et al., "Update Notification Filtering Method Related to Information Security", Institute of Electronics, Information and Communication Engineers, 2024 General Conference, B-14-11 (February 26, 2024)

[0004] Many of the sentences in the well-known document are composed of stereotyped sentences. Here, the parts that should be particularly made known to the user are often the non-stereotyped differential parts. Therefore, it is beneficial to present the differences between the well-known document and other well-known documents to the user. However, since the sentences in the well-known document are often stereotyped sentences, when many well-known documents with different content to be made known are mixed, differences may be extracted between well-known documents with different content to be made known. Even if the differences between well-known documents with significantly different content to be made known are presented to the user, it cannot necessarily be said that it is beneficial to the user.

[0005] The embodiment provides a document processing device capable of appropriately extracting differences between documents.

[0006] One embodiment of a document processing device comprises a metadata assignment unit, a clustering unit, a difference extraction unit, and an output unit. The metadata assignment unit vectorizes multiple documents based on the number of occurrences of words contained in each document, and assigns the vectors obtained by vectorization and the publication time of each document as metadata to the corresponding documents. The clustering unit clusters the documents based on the similarity of the number of occurrences of words represented by the vectors assigned by the metadata assignment unit. The difference extraction unit sorts the documents in order of publication time for each cluster formed by the clustering unit, and extracts the differences between the documents sorted in order of publication time. The output unit outputs the differences between documents.

[0007] According to one embodiment, a document processing device is provided that is capable of appropriately extracting differences between documents.

[0008] Figure 1 is a block diagram showing the configuration of a document processing device according to an embodiment. Figure 2 is a conceptual diagram illustrating the assignment of metadata in the embodiment. Figure 3 is a conceptual diagram illustrating clustering in the embodiment. Figure 4 is a conceptual diagram illustrating difference extraction in the embodiment. Figure 5 is a flowchart showing the operation of the document processing device according to an embodiment. Figure 6 is a diagram showing an example of a well-known document acquired by the metadata assignment unit. Figure 7 is a diagram showing an example of clustering for well-known documents D1, D2, D3, and D4. Figure 8 is a diagram showing a specific example of difference extraction. Figure 9 is a block diagram showing an example of the hardware configuration of the document processing device.

[0009] Embodiments will be described below with reference to the drawings. Figure 1 is a block diagram showing the configuration of a document processing apparatus according to an embodiment.

[0010] The document processing device according to the embodiment extracts differences between known documents and presents the extracted differences to the user. In the embodiment, the known document is a document that contains information that should be made known to the user. The known document is, for example, an update notification document that notifies the user to perform a software update. Since the update notification document is also a document that instructs the user to perform the update, it can also be called an instruction document. The known document may be data in any format that includes a notification text, such as data in the body of an email or document data attached as an attachment.

[0011] The document processing device 1 includes a metadata assignment unit 11, a clustering unit 12, a difference extraction unit 13, and an output unit 14.

[0012] The metadata assignment unit 11 assigns metadata to the input publicly known document D. Metadata is information that indicates the content of the data. In this embodiment, the metadata includes an occurrence count vector and the publication time. The metadata assignment unit 11 vectorizes the text of the input publicly known document D based on the number of occurrences of each word. Vectorization based on the number of occurrences of each word can be performed using methods such as BOW (Bag of Words). The metadata assignment unit 11 then associates the generated vector and the publication time of the publicly known document D with the corresponding publicly known document D as metadata.

[0013] Figure 2 is a conceptual diagram illustrating the addition of metadata in the embodiment. The metadata addition unit 11, for example, divides the text of the well-known document D into units of words by morphological analysis and counts the number of occurrences of each divided word. The metadata addition unit 11 then generates a vector whose elements are the number of occurrences of each word. Furthermore, the metadata addition unit 11 associates the word occurrence vector V of the well-known document D with the publication time t of the well-known document D. In other words, for each well-known document D, the metadata addition unit 11 generates data with two attributes: the word occurrence vector V and the publication time t of the well-known document.

[0014] Here, a single notification document D typically contains text that indicates the content of one notification. However, it is also conceivable that a notification document D may contain text that indicates multiple different types of notifications. If a notification document D contains text that indicates multiple different types of notifications, the metadata assignment unit 11 vectorizes each text that indicates the content of the notification as a separate entity. In the following process, notification documents to which separate vectors have been assigned are treated as separate notification documents.

[0015] The clustering unit 12 clusters the multiple known documents D based on the similarity of the vector V of word occurrences assigned to each of the known documents D.

[0016] Figure 3 is a conceptual diagram illustrating clustering in an embodiment. The clustering unit 12 classifies the known documents D into clusters based on the similarity between vectors V obtained by methods such as calculating the Euclidean distance between vectors V for each known document D or calculating cosine similarity. In Figure 3, the number of word occurrences in known documents 1, 3, and 5 are similar, and the number of word occurrences in known documents 2 and 4 are also similar. Therefore, the clustering unit 12 classifies known documents 1, 3, and 5 into one cluster and known documents 2 and 4 into another cluster. Cluster labels may be arbitrarily assigned as long as they can identify the clusters. Figure 3 shows an example in which known documents 1, 3, and 5 are classified into cluster (1), and known documents 2 and 4 are classified into cluster (2). The labels assigned to the clusters may be set as appropriate.

[0017] The difference extraction unit 13 extracts the differences between a plurality of public documents D. In this embodiment, the difference extraction unit 13 rearranges the public documents belonging to the same cluster in chronological order and extracts the differences between them and the previous public document in chronological order. The differences include differences in words and symbols. The previous public document in chronological order is, for example, the public document immediately preceding it in chronological order. The previous public document in chronological order may be two or more public documents preceding it in chronological order.

[0018] Figure 4 is a conceptual diagram illustrating the extraction of differences in the embodiment. Here, in Figure 4, the time interval t1-t14 is, for example, one day, but is not limited to one day.

[0019] The difference extraction unit 13 arranges the public notices belonging to each cluster in order of publication time. The difference extraction unit 13 then calculates the difference between public notices whose publication times are adjacent, as indicated by the arrows in Figure 4. The difference can be the difference between each element of vector V. The difference extraction unit 13 then divides the difference vector into positive-dimensional elements and negative-dimensional elements. The positive-dimensional elements mean that there are words that have been added compared to the previous public notice. The negative-dimensional elements mean that there are words that have been removed compared to the previous public notice. Finally, the difference extraction unit 13 converts the negative-dimensional elements and the positive-dimensional elements back into the words or symbols that represent their corresponding occurrence counts. The words or symbols converted back from the negative-dimensional elements and the words or symbols converted back from the positive-dimensional elements are the differences between the public notices D.

[0020] The output unit 14 performs various output processing based on the differences extracted by the difference extraction unit 13. For example, the output unit 14 displays on the display a list of words that have been added and removed from the previous public notice document, as extracted by the difference extraction unit 13. Alternatively, the output unit 14 transmits information about the words that differ between each public notice document D and the previous public notice document to an external device of the document processing device 1.

[0021] Next, the operation of the document processing device 1 will be described. Figure 5 is a flowchart showing the operation of the document processing device 1 according to this embodiment. The operation in Figure 5 is performed at regular intervals such as one day, one week, one month, or one year. Alternatively, the operation in Figure 5 may be performed in response to instructions from the user.

[0022] In step S1, the metadata assignment unit 11 of the document processing device 1 acquires a plurality of publicly known documents D and assigns metadata to each of the acquired publicly known documents D. Figure 6 is a diagram showing an example of publicly known documents acquired by the metadata assignment unit 11. In Figure 6, the metadata assignment unit 11 acquires publicly known documents D1, D2, D3, and D4. Among the publicly known documents D1, D2, D3, and D4, publicly known document D4 is the newest and publicly known document D1 is the oldest. Publicly known documents D1, D2, D3, and D4 may be pre-stored in a storage unit (not shown) of the document processing device 1, or they may be input by a user. For each of the publicly known documents D1, D2, D3, and D4, the metadata assignment unit 11 counts the number of occurrences of words and generates a vector with the number of occurrences of each word as an element. Then, the metadata assignment unit 11 associates the vector of word occurrences as metadata with the publication time for the publicly known documents D1, D2, D3, and D4. In Figure 6, the metadata assignment unit 11 acquires four publicly known documents. However, the number of publicly known documents acquired by the metadata assignment unit 11 is not limited to four.

[0023] In step S2, the clustering unit 12 clusters the known documents D based on the vector of word occurrences assigned to each known document D. Figure 7 shows an example of clustering for known documents D1, D2, D3, and D4. In the example shown in Figure 6, the word occurrences of known documents D1 and D4 are similar. Therefore, the clustering unit 12 clusters known documents D1 and D4 into the same cluster, cluster (2) in Figure 7. On the other hand, the word occurrences of known documents D2 and D3 are not similar to those of known documents D1 and D4. Also, the word occurrences of known documents D2 and D3 are not similar. Therefore, the clustering unit 12 clusters known documents D2 and D3 into separate clusters (1) and (3), respectively.

[0024] In step S3, the difference extraction unit 13 rearranges the public documents D in chronological order for each cluster. For example, the difference extraction unit 13 rearranges the public documents D1 and D4, which are clustered in cluster (2), in the order of public documents D1 and D4, which is the order of their issuance times.

[0025] In step S4, the difference extraction unit 13 calculates the difference between the public documents D for each cluster. Specifically, the difference extraction unit 13 calculates the element-wise difference between the first vector assigned to the first public document and the second vector assigned to the second public document, which is issued immediately before the first public document in chronological order. The difference extraction unit 13 then extracts a difference vector consisting of difference elements whose values ​​are greater than or equal to a threshold. For example, as shown in Figure 8, the difference extraction unit 13 calculates the difference between public document D4 and public document D1. As a difference between public document D4 and public document D1, for example, the issue date "January 30, 2024" in public document D1 is "February 6, 2024" in public document D4. In other words, the element corresponding to "February 6, 2024" in public document D4 is an element that does not exist in public document D1, so when the difference is taken, it becomes a large value. Furthermore, the element corresponding to "January 30, 2024" in Notification Document D1 is not present in Notification Document D4, so the difference will be large. Similarly, "ver. x1," which specifies the version of the software to be updated in Notification Document D1, is "ver. x2" in Notification Document D4. In other words, the element corresponding to "ver. x2" in Notification Document D4 is not present in Notification Document D1, so the difference will be large. Also, the element corresponding to "ver. x1" in Notification Document D1 is not present in Notification Document D4, so the difference will be large. Similarly, Notification Document D1 states "Request to perform the update," while Notification Document D4 states "Please postpone the update." In other words, the element corresponding to "postpone" in Notification Document D4 is not present in Notification Document D1, so the difference will be large. Furthermore, the element corresponding to "implementation" in notification document D1 is an element that does not exist in notification document D4, so the difference will be a large value.Based on the above, the difference extraction unit 13 extracts a difference vector that includes as elements the difference for the element corresponding to "January 30, 2024", the difference for the element corresponding to "February 6, 2024", the difference for the element corresponding to "ver. x1", the difference for the element corresponding to "ver. x2", the difference for the element corresponding to "implemented", and the difference for the element corresponding to "held".

[0026] In step S5, the difference extraction unit 13 divides the extracted difference vector into positive-dimensional elements and negative-dimensional elements. The positive-dimensional elements are elements with a positive difference value. The negative-dimensional elements are elements with a negative difference value. In the example in Figure 8, the positive elements are the elements corresponding to "February 6, 2024", the elements corresponding to "ver. x2", and the elements corresponding to "Pending". On the other hand, the negative elements are the elements corresponding to "January 30, 2024", the elements corresponding to "ver. x1", and the elements corresponding to "Implemented". The difference extraction unit 13 then returns the elements of the vector divided into positive elements and the elements of the vector divided into negative elements back to the words or symbols that represent the corresponding occurrence counts. The difference extraction unit 13 then outputs a list L containing the words or symbols returned from the positive elements and the words or symbols returned from the negative elements.

[0027] In step S6, the output unit 14 performs various output processing based on the differences extracted by the difference extraction unit 13. After that, the processing shown in Figure 5 is completed. For example, the output unit 14 displays on the display a list of words that have been added and removed from the previous notification document, as extracted by the difference extraction unit 13. Specifically, it displays the list L shown in Figure 8 for each cluster and for each notification document. Alternatively, the output unit 14 transmits the information of the words that are the difference between each notification document D and the previous notification document to an external device of the document processing device 1.

[0028] Figure 9 is a block diagram showing an example of the hardware configuration of the document processing device 1. As shown in Figure 9, the document processing device 1 includes, for example, a processor 101, a ROM (read-only memory) 102, a RAM (random access memory) 103, a communication module 104, a display 105, and storage 106. The document processing device 1 may be, for example, a terminal device such as a smartphone or a PC (personal computer). In addition, the document processing device 1 may be a device built on a cloud environment.

[0029] The processor 101 is a processing circuit that controls the overall operation of the document processing device 1. The processor 101 operates as a metadata assignment unit 11, a clustering unit 12, a difference extraction unit 13, and an output unit 14 by executing a document processing program stored in the storage 106, for example. The processor 101 is, for example, a CPU. The processor 101 may also be an MPU, GPU, ASIC, FPGA, etc. The processor 101 may be a single CPU, or multiple CPUs, etc.

[0030] ROM 102 is a non-volatile semiconductor memory. ROM 102 stores the startup program and various setting values ​​of the document processing device 1. RAM 103 is a volatile memory. RAM 103 is used, for example, as working memory during processing in the processor 101.

[0031] The communication module 104 is a circuit equipped with an interface for the document processing device 1 to communicate with external devices. The communication module 104 may be a wireless communication module or a wired communication module.

[0032] The display 105 is, for example, an LCD (liquid crystal display) or an EL (electro-luminescence) display. The display 105 displays a GUI (graphical user interface) or the like for various applications such as email applications.

[0033] The storage device 106 is a non-volatile storage device such as a hard disk or flash memory. The storage device 106 stores various programs, such as the operating system (OS) and document processing programs of the document processing device 1. The document processing programs may be downloaded to the storage device 106 via a network.

[0034] As described above, according to the embodiment, known documents are clustered based on the similarity of the number of occurrences of words in the known documents, and differences are extracted from the known documents that have been rearranged in chronological order for each cluster. Known documents clustered based on the similarity of the number of occurrences of words are likely to be known documents about similar content. Therefore, in this embodiment, even among known documents that contain a mixture of different content, differences between known documents about similar content can be appropriately extracted.

[0035] Furthermore, in recent years, attempts have been made to classify publicly known documents using machine learning models. This classification can utilize the differences and commonalities between publicly known documents. In this embodiment, since the differences between publicly known documents with similar content are extracted, it is expected that appropriate differences and commonalities will be input to the machine learning model.

[0036] In this embodiment, the publicly known document subject to difference extraction is defined as an updated publicly known document. However, the publicly known document subject to difference extraction is not limited to updated publicly known documents. That is, the technology of this embodiment can be applied to any document other than updated publicly known documents, as long as the documents are issued in chronological order.

[0037] It should be noted that the present invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriate, and in that case, the combined effects can be obtained. Moreover, the above embodiments include various inventions, and various inventions can be extracted by selecting combinations from the multiple constituent elements disclosed. For example, if the problem can be solved and effects obtained even if some constituent elements are deleted from all the constituent elements shown in the embodiment, then the configuration with these deleted constituent elements can be extracted as an invention.

[0038] 1...Document processing device 11...Metadata assignment unit 12...Clustering unit 13...Difference extraction unit 14...Output unit 101...Processor 102...ROM 103...RAM 104...Communication module 105...Display 106...Storage

Claims

1. A document processing device comprising: a metadata assignment unit that vectorizes multiple documents based on the number of occurrences of words contained in each document, and assigns the vectors obtained by vectorization and the publication time of each document as metadata to the corresponding documents; a clustering unit that clusters the documents based on the similarity of the number of occurrences of words represented by the vectors assigned by the metadata assignment unit; a difference extraction unit that sorts the documents in order of publication time for each cluster clustered by the clustering unit, and extracts the differences between the documents sorted in order of publication time; and an output unit that outputs the differences between the documents.

2. The document processing apparatus according to claim 1, wherein the difference extraction unit calculates the difference between each element of the vector between the first document and the second document among the documents sorted in order of publication time, divides the calculated difference elements into positive dimension elements where the difference value is positive and negative dimension elements where the difference value is negative, and returns the positive dimension elements and the negative dimension elements to corresponding words, respectively.

3. The document processing apparatus according to claim 2, wherein the output unit displays on a display a list including the words returned from the positive-dimensional elements and the words returned from the negative-dimensional elements.

Images