Document data processing device, document data processing method, and program

Abstract

Even if the document data has not been pre-categorized, the system will determine the similarity between the document data and extract similar document data. [Solution] The system includes: acquisition means for acquiring document data having one or more items; word extraction means for extracting words from string data indicated in the items of the document data; vectorization means for vectorizing each of the string data based on the words in each of the string data; similarity calculation means for calculating the similarity between string data having the same item based on the vectors corresponding to each of the string data; and grouping means for selecting document data that have a direct and indirect linking relationship based on the similarity and a predetermined similarity threshold, and grouping each of the selected combinations with the linking relationship.

Description

Technical Field

[0001] The present disclosure relates to a document data processing apparatus, a document data processing method, and a program.

Background Art

[0002] In Patent Document 1, a system is disclosed that supports finishing a final answer in a mail call center by the system presenting appropriate answer candidates for questions, selecting some of the answer candidates for proofreading, and thus finishing the final answer. In order to narrow down and show the answer candidates that are the basis of this proofreading, in the technique disclosed in Patent Document 1, by using document data including a question sentence obtained in the past and an answer sentence corresponding to the question sentence, and vectorizing the question sentences included in the document data, similar document data is detected.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the technique disclosed in Patent Document 1 assumes that the document data obtained in the past is pre-classified into categories. The vector representing the category is calculated as an average feature vector, and the similarity of the document data is determined by comparison with the calculated average feature vector. Therefore, the technique disclosed in Patent Document 1 has a problem that the similarity between document data not pre-classified into categories cannot be determined.

[0005] In contrast, the amount of document data, including questions and answers, obtained from contact centers and call centers (hereinafter referred to as "contact centers, etc.") is enormous, and classifying this vast amount of document data into categories is extremely time-consuming. Therefore, there is a need to be able to determine the similarity between document data and extract similar document data, even if the document data has not been pre-classified.

[0006] The purpose of this disclosure is to provide a document data processing device, a document data processing method, and a program that solve the above-mentioned problems. [Means for solving the problem]

[0007] A document data processing device according to one aspect of the present disclosure includes: acquisition means for acquiring document data having one or more items; word extraction means for extracting words from string data indicated in the items of the document data; vectorization means for vectorizing each of the string data based on the words in each of the string data; similarity calculation means for calculating the similarity between string data having the same items based on the vectors corresponding to each of the string data; and grouping means for selecting document data that have a direct and indirect linking relationship based on the similarity and a predetermined similarity threshold, and grouping each of the selected combinations having the linking relationship.

[0008] A document data processing method according to one aspect of the present disclosure involves acquiring document data having one or more items, extracting words from string data indicated in the items of the acquired document data, vectorizing each of the string data based on the extracted words, calculating the similarity between string data having the same items based on the vectors corresponding to each of the string data, selecting document data that have direct and indirect relationships based on the calculated similarity and a predetermined similarity threshold, and grouping each of the selected combinations with such relationships.

[0009] A program according to one aspect of this disclosure is a program for causing a computer to function as: an acquisition means for acquiring document data having one or more items; a word extraction means for extracting words from string data indicated in the items of the document data; a vectorization means for vectorizing each of the string data based on the words in each of the string data; a similarity calculation means for calculating the similarity between string data having the same items based on the vectors corresponding to each of the string data; and a grouping means for selecting document data that have a direct and indirect association relationship based on the similarity and a predetermined similarity threshold, and grouping each of the selected combinations with the association relationship. [Effects of the Invention]

[0010] According to the above embodiment, even if the document data has not been pre-classified into categories, the similarity between the document data can be determined and similar document data can be extracted. [Brief explanation of the drawing]

[0011] [Figure 1] This block diagram shows an example of the configuration of a document data processing device related to this disclosure. [Figure 2] This figure shows an example of the response data related to this disclosure. [Figure 3] This figure shows an example of the procedure for word extraction by the word extraction unit related to this disclosure. [Figure 4] This figure shows an example of community detection by the community detection unit related to this disclosure. [Figure 5] This flowchart shows an example of the processing flow by the document data processing device related to this disclosure. [Figure 6] This figure shows an example of vector data generated by the vectorization unit relating to this disclosure. [Figure 7] This diagram illustrates the combinations of correspondence data used for calculating similarity in this disclosure. [Figure 8]This is a diagram showing an example of the procedure for calculating similarity by the similarity calculation unit according to the present disclosure. [Figure 9] This is a diagram showing an example of a transition relationship consideration group according to the present disclosure. [Figure 10] This is a diagram showing an example of the affiliated group data according to the present disclosure. [Figure 11] This is a diagram showing an example of the response data according to the present disclosure. [Figure 12] This is a diagram showing an example of the group representative response data according to the present disclosure. [Figure 13] This is a diagram showing an example of the response data compressed by the group representative response data according to the present disclosure. [Figure 14] This is a diagram showing an example of the procedure for calculating the contribution degree by the group feature analysis unit according to the present disclosure. [Figure 15] This is a diagram showing an example of the analysis result displayed on the screen of the output device by the group feature analysis unit according to the present disclosure. [Figure 16] This is a block diagram showing an example of the configuration of the document data processing device according to the present disclosure. [Figure 17] This is a flowchart showing an example of the processing flow by the document data processing device according to the present disclosure. [Figure 18] This is a diagram for explaining the combination of the response data for which similarity is to be calculated according to the present disclosure. [Figure 19] This is a diagram showing the combination pattern between the incremental response data and the group by the group combination adjustment unit according to the present disclosure. [Figure 20] This is a diagram showing an example of the combination relationship between the existing transition relationship consideration group and the incremental response data according to the present disclosure. [Figure 21] This is a diagram showing an example of the transition relationship consideration group after the adjustment by the group combination adjustment unit according to the present disclosure. [Figure 22] This is a block diagram showing an example of the configuration of the document data processing device according to the present disclosure. [Figure 23] This is a flowchart showing an example of the processing flow by the document data processing device when generating the co-occurrence word data according to the present disclosure. [Figure 24] This diagram illustrates the procedure for generating co-occurring word data related to this disclosure. [Figure 25] This figure shows an example of co-occurring word data related to this disclosure. [Figure 26] This flowchart shows an example of the processing flow by a document data processing device when displaying co-occurring words related to this disclosure. [Figure 27] This figure shows an example of a search query input screen displayed by the search query acquisition unit related to this disclosure. [Figure 28] This diagram illustrates the process by which the co-occurring word recommendation unit related to this disclosure selects the co-occurring words to display. [Figure 29] This figure shows an example of a search query input screen displayed by the search query acquisition unit related to this disclosure. [Figure 30] This diagram illustrates the process by which the co-occurring word recommendation unit related to this disclosure selects the co-occurring words to display. [Figure 31] This is a block diagram showing the hardware configuration of the document data processing device related to this disclosure. [Figure 32] This block diagram shows an example of the configuration of a document data processing device related to this disclosure. [Figure 33] This flowchart shows an example of the processing flow by the document data processing device related to this disclosure. [Modes for carrying out the invention]

[0012] Each embodiment will be described below with reference to the drawings. In all drawings, identical or equivalent components are denoted by the same reference numerals, and common descriptions are omitted.

[0013] <First Embodiment> Hereinafter, an embodiment relating to this disclosure will be described with reference to the figures. As shown in Figure 1, the response data storage unit 2 stores multiple response data. The response data storage unit 2 stores, for example, a table having the items "Case ID (Identification)", "Question", "Answer", and "Other" as shown in Figure 2, and each record in each row of the table becomes an individual response data.

[0014] The "Case ID" field records information that makes each interaction data unique, such as a case ID that allows each interaction data to be identified by a different number. The "Question" field records the string data of the question text, which indicates the content of the question received by the questioner, for example, in a contact center. The "Answer" field records the string data of the answer text, which indicates the content of the answer given by the questioner to the questioner. The "Other" field records information such as the name of the product that the questioner inquired about.

[0015] The document data processing device 1 comprises an acquisition unit 11, a word extraction unit 12, a vectorization unit 13, a similarity calculation unit 14, a grouping unit 15, an intermediate data storage unit 20, a group representative data generation unit 31, and a group feature analysis unit 32. The acquisition unit 11 is connected to the response data storage unit 2, for example, via a wired or wireless line, and acquires the response data stored in the response data storage unit 2.

[0016] The word extraction unit 12 acquires the string data of the question sentence shown in the "Question" item and the string data of the answer sentence shown in the "Answer" item of the response data acquired by the acquisition unit 11, and extracts the words contained in each of the acquired string data using a method such as morphological analysis. Once the word extraction unit 12 has extracted words from each of the response data, it generates word data having the following data format from the extracted words. The word extraction unit 12 generates word data for each response data, as word data, associated with the case ID shown in the "Case ID" item of the response data, and including words extracted from the string data of the question sentence corresponding to the case ID (hereinafter referred to as the question sentence word group) and words extracted from the string data of the answer sentence corresponding to the case ID (hereinafter referred to as the answer sentence word group).

[0017] Figure 3 shows an example of the word extraction procedure performed by the word extraction unit 12. For example, if the string data is "I am going on a business trip now," the word extraction unit 12 performs morphological analysis on this string data to divide it into words. The word extraction unit 12 analyzes the part of speech of each word divided by morphological analysis. Based on the part of speech information of each word obtained as a result of the part of speech analysis, the word extraction unit 12 deletes words of predetermined parts of speech, and removes the part of speech information from the remaining words to output the words "I," "this," "business trip," and "going" as a word group. In Figure 3, an example is shown where the parts of speech to be deleted are parts of speech other than "pronouns," "nouns," and "verbs," but the parts of speech to be deleted can be arbitrarily determined. Alternatively, the word extraction unit 12 may output all words obtained by morphological analysis as a word group without performing part of speech analysis.

[0018] The vectorization unit 13 vectorizes the string data of the question and answer sentences for each response data based on the word sets of the question and answer sentences for each word data generated by the word extraction unit 12. Hereinafter, the vector generated based on the word set of the question sentence will be called the question sentence vector, and the vector generated based on the word set of the answer sentence will be called the answer sentence vector. The vectorization unit 13 generates vector data for each word data by associating the case ID contained in the word data with the question sentence vector corresponding to the case ID and the answer sentence vector corresponding to the case ID. As the vectorization method performed by the vectorization unit 13, for example, the TF-IDF (Term Frequency-Inverse Document Frequency) method is applied. Note that other arbitrary methods such as TF (Term Frequency), BM25 (Best Matching 25), Word2Vec, Doc2Vec, etc., other than TF-IDF may be applied.

[0019] The similarity calculation unit 14 calculates the similarity between response data for all combinations of two response data selected from multiple response data, based on the vector data generated by the vectorization unit 13. Specifically, the similarity calculation unit 14 calculates the similarity between the vectors within the same item corresponding to each of the two response data for each combination of response data, i.e., the similarity between the question vectors and the similarity between the answer vectors. As the similarity calculation method used by the similarity calculation unit 14, for example, the cosine similarity method is applied. However, other arbitrary methods such as the Euclidean norm may also be applied. The similarity calculation unit 14 generates similarity data for each combination of two response data, which includes the combination of case IDs corresponding to the combination of two response data, the similarity between the question vectors corresponding to that combination, and the similarity between the answer vectors.

[0020] The grouping unit 15 comprises a high-similarity data selection unit 16, a transition relationship-considering group determination unit 17, and a community detection unit 18. The high-similarity data selection unit 16 selects a combination of two response data that has a high similarity relationship based on the similarity shown by the similarity data generated by the similarity calculation unit 14 for a combination of two response data and a predetermined similarity threshold.

[0021] The Transition Relationship Consideration Group Determination Unit 17 identifies two pairs of response data with a high similarity relationship, selected by the High Similarity Data Selection Unit 16, as two pairs of response data with a direct connection relationship. The Transition Relationship Consideration Group Determination Unit 17 further selects response data with an indirect connection relationship, and determines each of the selected combinations of response data with direct and indirect connections as a Transition Relationship Consideration Group. The Transition Relationship Consideration Group Determination Unit 17 generates Transition Relationship Consideration Group data by associating the case ID of the response data belonging to each Transition Relationship Consideration Group, information indicating the connection relationship between the response data, and the similarity of the question and answer sentences corresponding to that connection relationship, with respect to the Transition Relationship Consideration Group Identification Information that can identify each Transition Relationship Consideration Group.

[0022] The community detection unit 18 divides each of the transition relationship consideration groups by performing community detection on each of the transition relationship consideration groups based on the transition relationship consideration group data generated by the transition relationship consideration group determination unit 17. The community detection unit 18 determines each of the divided groups as the group to which the response data belongs. The community detection unit 18 is connected to the response data storage unit 2, for example, via a wired or wireless line, and obtains string data from the response data storage unit 2 that are shown in the "Question," "Answer," and "Other" items of the response data corresponding to the case ID of all the response data belonging to each of the group to which the group belongs. The community detection unit 18 generates group to which data belongs by associating the case ID of the response data belonging to any of the group to which the case ID belongs with the string data shown in the "Question," "Answer," and "Other" items of the response data corresponding to the case ID and the group to which the response data corresponding to the case ID belongs with group identification information.

[0023] The community detection performed by the community detection unit 18 is a clustering method in graph network analysis, and for example, a greedy algorithm is applied. However, other arbitrary methods such as edge-between centrality-based methods or random walk-based methods may also be applied. For example, suppose the graph represented by a certain transitional relationship group determined by the transitional relationship group determination unit 17 has the shape shown in Figure 4. In Figure 4, each of the 11 symbols—a black and white circle, triangle, square, rhombus, and inverted triangle, and an X mark—represents individual correspondence data belonging to that transitional relationship group. As shown in Figure 4, if the graph represented by a single transitional relationship group becomes too large, the ends of the graph may have different meanings. In this case, the community detection unit 18 divides the response data into 11 groups, represented by black and white circles, triangles, squares, diamonds, and inverted triangles, and cross marks, based on the shape of the graph of the transition relationship consideration group through community detection. Each of the 11 divided groups becomes the member group.

[0024] The group representative data generation unit 31 generates group representative response data for each affiliated group from the response data belonging to each affiliated group, based on the affiliated group data generated by the community detection unit 18.

[0025] The group feature analysis unit 32 calculates the centroid vector for each affiliated group based on the affiliated group data generated by the community detection unit 18, using the vector data of the response data belonging to the affiliated group. Based on the calculated centroid vectors, the group feature analysis unit 32 calculates the distance between affiliated groups and the contribution of words common to the affiliated groups. Based on the distance between affiliated groups and the contribution of words common to the affiliated groups, the group feature analysis unit 32 performs analysis processing to draw a structural diagram of the affiliated groups and extract words common to the affiliated groups and characteristic words of each affiliated group.

[0026] The intermediate data storage unit 20 stores word data generated by the word extraction unit 12 for each case ID, vector data generated by the vectorization unit 13 for each case ID, transition relationship-considering group determination unit 17, belonging group data generated by the community detection unit 18, and group representative response data generated by the group representative data generation unit 31.

[0027] Output device 3 is, for example, a display device such as a liquid crystal display, and is connected to the group feature analysis unit 32. Output device 3 displays the results of the analysis processing performed by the group feature analysis unit 32, etc.

[0028] (Grouping of response data by document data processing device 1) The following describes the process by which the document data processing device 1 groups response data, referring to the flowchart shown in Figure 5. The acquisition unit 11 acquires response data one by one from the response data storage unit 2 and outputs the acquired response data to the word extraction unit 12. When the acquisition unit 11 has acquired all the response data stored in the response data storage unit 2 and finished outputting to the word extraction unit 12, it outputs a completion notification signal to the word extraction unit 12 (Sa1).

[0029] Each time the word extraction unit 12 takes in the response data output by the acquisition unit 11, it extracts words from the string data of the "question" item and the string data of the "answer" item of the acquired response data according to the procedure described above. In the following example, the part of speech predetermined to be deleted by the word extraction unit 12 is a part of speech other than "noun". The word extraction unit 12 generates word data containing the extracted words for each response data and records the generated word data in the intermediate data storage unit 20. The word extraction unit 12 receives a completion notification signal from the acquisition unit 11, and if the generation and recording of word data corresponding to all acquired response data has been completed by the time the completion notification signal is received, it outputs the completion notification signal to the vectorization unit 13 (Sa2).

[0030] When the vectorization unit 13 receives a completion notification signal from the word extraction unit 12, it vectorizes all the word data stored in the intermediate data storage unit 20. For example, suppose the word extraction unit 12 extracts three words, "BIOS," "settings," and "how-to," from the string data "I don't know how to set the BIOS," shown in the "Question" item of the response data for case ID "2" in Figure 2. In this case, the word data generated by the word extraction unit 12 includes "BIOS," "settings," and "how-to" as the words of the question. Therefore, the vectorization unit 13 calculates the values ​​"0.65012," "0.15678," and "0.00110" for each of the words "BIOS," "settings," and "how-to" using the TF-IDF method. The values ​​calculated by the vectorization unit 13 are listed as element values ​​to form a vector of the string data of the question for case ID "2." When the vectorization unit 13 obtains a question vector corresponding to the group of words in the question sentence and an answer vector corresponding to the group of words in the answer sentence contained in a single word data, it associates the question vector and the answer vector with the case ID contained in the word data and generates vector data. The vectorization unit 13 records the generated vector data in the intermediate data storage unit 20.

[0031] Figure 6 shows a portion of the vector data stored in the intermediate data storage unit 20, and displays two vector data obtained from the two correspondence data for case IDs "2" and "3" shown in Figure 2. The first section, separated by " / ", shows the case ID. Therefore, the vector data enclosed in the dotted frame in the upper part of Figure 6 corresponds to the vector data for case ID "2", and the vector data enclosed in the dotted frame in the lower part corresponds to the vector data for case ID "3".

[0032] The second section, separated by " / ", shows the question vector, and the third section shows the answer vector. The string data shown in the "Question" item of the response data for case ID "3" shown in Figure 2 is "I don't know how to set the BIOS," and the word set of the question extracted by the word extraction unit 12 consists of two words: "BIOS" and "settings." Therefore, the second section shows the question vector as the word set of the question (BIOS, settings) and (0.61234, 0.20123). Note that "0.61234" is the element value corresponding to the word "BIOS," and "0.20123" is the element value corresponding to the word "settings." The string data shown in the "Answer" item of the response data for case IDs "2" and "3" is the same, "Please check page 9 of the manual," so the answer vectors for case IDs "2" and "3" shown in Figure 6 are the same.

[0033] When the vectorization unit 13 has finished generating and recording vector data corresponding to all word data stored in the intermediate data storage unit 20, it outputs a completion notification signal to the similarity calculation unit 14 (Sa3).

[0034] When the similarity calculation unit 14 receives a completion notification signal from the vectorization unit 13, it selects a combination of two vector data from the vector data stored in the intermediate data storage unit 20 and calculates the similarity between the vector data of the selected combination. If there are N response data (where N is an integer of 2 or more) stored in the response data storage unit 2, the total number of combinations is: N There are two Cs, and their combination corresponds to the white grid area in the relationship between the two response data shown in Figure 7. In the notation "Response data [n] (where n is an integer between 1 and N)" in Figure 7, "n" is the number representing the case ID.

[0035] The similarity calculation unit 14 sequentially reads out pairs of vector data corresponding to the white grid area in Figure 7 from the intermediate data storage unit 20. The similarity calculation unit 14 calculates the similarity between each question vector of the two pairs of vector data read out. For example, the procedure by which the similarity calculation unit 14 calculates the cosine similarity between the two question vectors of case IDs "2" and "3" shown in Figure 6 will be explained with reference to Figure 8. As shown in Figure 6, the question vector for case ID "2" is (BIOS, setting, how) (0.65012, 0.15678, 0.00100), and the question vector for case ID "3" is (BIOS, setting) (0.61234, 0.20123).

[0036] When the two question vectors are represented in a table format, with common words in one column and the words contained in the two question vectors arranged in the column direction, the table shown in Figure 8 is obtained. Here, the question vector for case ID "2" contains the word "shikata," but the question vector for case ID "3" does not contain the word "shikata," so the element value of "shikata" in the question vector for case ID "3" is set to "0."

[0037] The similarity calculation unit 14 multiplies the values ​​of two column elements that share a common word. In the example shown in Figure 8, the word "BIOS" is multiplied by "0.65012 × 0.61234" to calculate "0.39809". Similarly, the similarity calculation unit 14 calculates "0.03154" for the word "setting" and "0" for the word "how". The similarity calculation unit 14 calculates the sum of the three calculated multiplied values, "0.42963", as the cosine similarity, i.e., the similarity between the question vectors.

[0038] The similarity calculation unit 14 calculates the similarity between each answer sentence vector of each combination of two read vector data using the same procedure as the procedure for calculating the similarity between question sentence vectors. The similarity calculation unit 14 generates similarity data that includes the combination of two case IDs, the calculated similarity between question sentence vectors, and the calculated similarity between answer sentence vectors. For example, in the example shown in Figure 8, the similarity data generated by the similarity calculation unit 14 includes [2,3] which represents the combination of two case IDs, "0.4963" which represents the calculated similarity between question sentence vectors, and the calculated similarity between answer sentence vectors. The similarity calculation unit 14 outputs the generated similarity data to the high-similarity data selection unit 16. When the similarity calculation unit 14 has generated similarity data for all combinations of two vector data and output it to the high-similarity data selection unit 16, it outputs a completion notification signal to the high-similarity data selection unit 16 (Sa4).

[0039] The high-similarity data selection unit 16, upon receiving the similarity data output by the similarity calculation unit 14, selects a combination of two correspondence data sets with a high similarity relationship based on the similarity indicated by the received similarity data and a predetermined similarity threshold.

[0040] The similarity data includes the similarity of the question and the similarity of the answer. Therefore, the high-similarity data selection unit 16 selects a combination of two response data by, for example, the following procedure. For example, the high-similarity data selection unit 16 selects a combination of two response data in which the similarity of the question corresponding to the combination of two response data is equal to or greater than the similarity threshold, and the similarity of the answer corresponding to the combination of two response data is also above the similarity threshold. Here, the similarity threshold corresponding to the question and the similarity threshold corresponding to the answer may be the same value predetermined, or different values ​​may be predetermined, for example, the similarity threshold corresponding to the question may be set to "0.4" and the similarity threshold corresponding to the answer may be set to "0.3".

[0041] Each time the high-similarity data selection unit 16 selects a combination of two correspondence data, it outputs the similarity data corresponding to that combination to the transition relationship-considering group determination unit 17. The high-similarity data selection unit 16 receives a completion notification signal from the similarity calculation unit 14, and further, once the process of selecting based on the similarity threshold, which is performed on each of the similarity data acquired up to the time of receiving the completion notification signal, is completed, it outputs a completion notification signal to the transition relationship-considering group determination unit 17 (Sa5).

[0042] The transition relationship consideration group determination unit 17 takes in the similarity data output by the high-similarity data selection unit 16. When the transition relationship consideration group determination unit 17 receives a completion notification signal from the high-similarity data selection unit 16, it selects response data that have a direct and indirect co-relationship based on the combination of two response data indicated by each of the similarity data taken in up to the time the completion notification signal was received. The transition relationship consideration group determination unit 17 determines each of the selected combinations of response data as a transition relationship consideration group.

[0043] For example, suppose the 12 combinations of case IDs included in each of the similarity data taken in by the transition relationship consideration group determination unit 17 are [2,3], [2,8], [2,5], [2,13], [2,19], [3,8], [3,13], [3,19], [5,19], [8,13], [8,19], and [13,19]. In this case, the transition relationship consideration group determination unit 17 determines the transition relationship consideration group shown in the graph in Figure 9 based on the 12 combinations of response data. In Figure 9, each node represents response data, and the "m" in the notation "Response Data [m]" shown within each node is a number indicating the case ID. The two numbers shown in the upper and lower rows between each node are the similarity of the question text in the upper row and the similarity of the answer text in the lower row. In this example, the similarity calculation unit 14 sets a similarity threshold of "0.4" for the question text and a similarity threshold of "0.3" for the answer text. Therefore, the similarity of the upper-level question texts between each node is "0.4" or higher, and the similarity of the lower-level answer texts is "0.3" or higher.

[0044] As shown in Figure 9, the transition relationship consideration group determination unit 17 determines a direct connection relationship for each of the 12 combinations of response data with case IDs [2,3], [2,8], [2,5], [2,13], [2,19], [3,8], [3,13], [3,19], [5,19], [8,13], [8,19], and [13,19] included in the similarity data. In contrast, for example, there is no direct connection relationship between the response data for case ID "8" and the response data for case ID "5," but the response data for case ID "8" has an indirect connection relationship with the response data for case ID "5" through the response data for case IDs "2" and "19." Therefore, the transition relationship consideration group determination unit 17 determines a single transition relationship consideration group that includes not only response data with direct connections but also response data with indirect connections.

[0045] The transition relationship consideration group determination unit 17 determines one or more transition relationship consideration groups from the acquired similarity data and generates transition relationship consideration group identification information that can identify each of the transition relationship consideration groups. The transition relationship consideration group determination unit 17 generates transition relationship consideration group data that associates each of the generated transition relationship consideration group identification information with the case ID of the response data belonging to each transition relationship consideration group, information indicating the coupling relationship between the response data, and the similarity of the question sentence and answer sentence corresponding to the coupling relationship. The transition relationship consideration group determination unit 17 records the generated transition relationship consideration group data in the intermediate data storage unit 20 and outputs a completion notification signal to the community detection unit 18 (Sa6).

[0046] When the community detection unit 18 receives a completion notification signal from the transition relationship consideration group determination unit 17, it performs the community detection described above on the transition relationship consideration group indicated by the transition relationship consideration group data stored in the intermediate data storage unit 20. The community detection unit 18 divides the transition relationship consideration group based on the community detection and determines each of the divided groups as the group to which the response data belongs. This group becomes the final group of the response data. The community detection unit 18 obtains the string data shown in the "Question," "Answer," and "Other" items of the response data corresponding to the case ID of all the response data belonging to each of the group belonging to the community detection unit 18 from the response data storage unit 2.

[0047] The community detection unit 18 generates group identification information that can identify each of the confirmed member groups. For each response data belonging to any member group, the community detection unit 18 generates member group data that associates the case ID of each response data with the string data shown in the "Question," "Answer," and "Other" items of the response data corresponding to the case ID and the group identification information of the member group to which the response data corresponding to the case ID belongs. The community detection unit 18 records the generated member group data in the intermediate data storage unit 20.

[0048] Figure 10 shows an example of the group affiliation data stored in the intermediate data storage unit 20. The data format of the group affiliation data is the same as the data format of the response data shown in Figure 2, but with the addition of a "group affiliation" item. The "group affiliation" item records the group affiliation identification information. Note that in the example shown in Figure 10, the record corresponding to the response data for case ID "1" shown in Figure 2 is not shown. This means that the response data for case ID "1" was not selected by the high-similarity data selection unit 16 and was not included in the transition relationship consideration group and the group affiliation group.

[0049] When the community detection unit 18 records the group affiliation data in the intermediate data storage unit 20, it outputs a completion notification signal to the group representative data generation unit 31 and the group feature analysis unit 32 (Sa7).

[0050] The group representative data generation unit 31 generates group representative response data for each affiliated group, for example, as follows: When the group representative data generation unit 31 receives a completion notification signal from the community detection unit 18, it refers to the affiliated group data stored in the intermediate data storage unit 20 and detects case IDs with matching affiliated group identification information for each affiliated group identification information. These detected case IDs with matching affiliated group identification information are hereinafter referred to as the same affiliated group case ID group. For each same affiliated group case ID group, the group representative data generation unit 31 reads vector data from the intermediate data storage unit 20 corresponding to the case IDs included in the same affiliated group case ID group. Based on the vector data read in accordance with the same affiliated group case ID group, the group representative data generation unit 31 calculates the centroid vector of the same affiliated group case ID group for each same affiliated group case ID group. The calculated centroid vector is the centroid vector of the affiliated group corresponding to the same affiliated group case ID group.

[0051] The vector data includes a question vector and an answer vector. Therefore, the group representative data generation unit 31 considers the composite vector of the question vector and the answer vector contained in one vector data as the vector represented by that vector data, and calculates the centroid vector based on the composite vector. To explain the composite vector of the question vector and the answer vector in more detail, for example, if the question vector is 100-dimensional and the answer vector is 100-dimensional, even if they are the same word, the question vector and the answer vector are considered to have different dimensions, and the composite vector is a vector having the respective components of the question vector and the answer vector, i.e., a 200-dimensional vector.

[0052] The group representative data generation unit 31 detects one vector among the vectors represented by each case ID in the group of case IDs belonging to the same group that is closest in distance to the centroid vector of that group of case IDs. Since this detected vector is closest in distance to the centroid vector, the response data corresponding to this vector can be said to be representative response data that best represents the characteristics of each of the response data of the case IDs belonging to the group of case IDs belonging to the same group.

[0053] The group representative data generation unit 31 detects a record corresponding to the case ID corresponding to the detected vector from the group data in the intermediate data storage unit 20, reads the contents of each item of the detected record, "Question," "Answer," "Other," and "Group Affiliation," rearranges the read contents in the order of "Group Affiliation," "Question," "Answer," and "Other," and generates group representative response data. Once the group representative data generation unit 31 has generated group representative response data for each group of case IDs belonging to the same group, it records the generated group representative response data in the intermediate data storage unit 20 (Sa8) and terminates the process.

[0054] For example, suppose the response data storage unit 2 stores response data for response case IDs "10" to "18" as shown in Figure 11. In this case, the group identification information for the group to which the response data for case IDs "10" to "13" belong becomes "20", the group identification information for the group to which the response data for case IDs "14" to "17" belong becomes "21", and the response data for case ID "18" is not classified into any group.

[0055] The group representative data generation unit 31 detects the response data for case ID "10" as group representative response data for the group case ID group of the same group identification information "20". Furthermore, the group representative data generation unit 31 detects the response data for case ID "14" as group representative response data for the group case ID group of the same group identification information "21". In this case, the group representative data generation unit 31 records the group representative response data shown in Figure 12 in the intermediate data storage unit 20.

[0056] By using this group representative response data, for example, the following can be done: In the response data storage unit 2, a case ID is detected in which the content of each of the "Question," "Answer," and "Other" items in the response data stored therein matches the content of each of the "Question," "Answer," and "Other" items in the record of the group identification information "20" of the group representative response data shown in Figure 12. In this case, case ID "10" is detected. Therefore, in the response data storage unit 2, the response data for case ID "10" is retained, and the response data for case IDs "11" to "13" that are classified under the group identification information "20" other than case ID "10" are deleted.

[0057] Similarly, the system detects a case ID in which the content of each item, "Question," "Answer," and "Other," matches the content of each item in the record of the group identification information "21" in the group representative response data shown in Figure 12. In this case, case ID "14" is detected. Therefore, the response data storage unit 2 keeps the response data for case ID "14" and deletes the response data for case IDs "15" to "17" that are classified in the group identification information "21" other than case ID "14."

[0058] This allows the nine records of the response data in the response data storage unit 2, with case IDs "10" to "18", to be compressed into three records, with case IDs "10", "14", and "18", as shown in Figure 13, while maintaining the characteristics of the response data belonging to the group identified by the group identification information "20" and "21".

[0059] Returning to Figure 5, when the group feature analysis unit 32 receives a completion notification signal from the community detection unit 18, it calculates the centroid vector of each of the group cases with the same belonging group IDs, i.e., the centroid vector of each belonging group, using the same procedure as described in the processing of Sa8 (Sa9). The group feature analysis unit 32 then calculates the distance between the calculated centroid vectors of each belonging group (Sa10-1).

[0060] The group feature analysis unit 32 calculates the contribution of common words between the member groups based on the centroid vectors of each member group calculated. For example, suppose the group feature analysis unit 32 calculates the first row of records shown in Figure 14 as the centroid vector of member group identification information "4", and calculates the second row of records shown in Figure 14 as the centroid vector of member group identification information "5". In the following, when referring to member group identification information "4", the term "identification information" will be omitted and it will also be referred to as member group "4", and in the diagram it will also be referred to as "member group 4". The same will be applied to other numbers other than "4".

[0061] The group feature analysis unit 32 multiplies the element values ​​of common words in the centroid vector of group 4 and the centroid vector of group 5. In this case, for example, if a word exists in one centroid vector but not in the other, such as the word "one-way call," the element value of the centroid vector where the word does not exist is set to "0" for the multiplication. The multiplied value obtained from this multiplication becomes the contribution of common words between group 4 and group 5. That is, as shown in Figure 14, the contribution of the word "call" between group 4 and group 5 is "0.30000," the contribution of the word "outgoing call" is "0.20000," the contribution of the word "outbound call" is "0.10000," and the contribution of the word "one-way call" is "0."

[0062] The group feature analysis unit 32 generates contribution data (Sa10-2) by associating each of the contributions calculated for a given combination of two group identification pieces that indicate the relationship between the groups to which the group belongs with the corresponding word.

[0063] The group feature analysis unit 32 generates an image as shown in Figure 15 based on the centroid vector calculated in the processing of Sa9, the distance between centroid vectors calculated in the processing of Sa10-1, the contribution data generated in the processing of Sa10-2, the affiliated group data, and the transition relationship-considering group data. The group feature analysis unit 32 detects groups of case IDs belonging to the same affiliated group from the affiliated group data. For each detected group of case IDs belonging to the same affiliated group, the group feature analysis unit 32 detects the connection relationships of the response data corresponding to the case IDs included in the group of case IDs belonging to the same affiliated group from the transition relationship-considering group data. Based on the detected connection relationships of the response data, the group feature analysis unit 32 generates a graph representing the connection relationships between the response data belonging to each affiliated group.

[0064] The group feature analysis unit 32 adds symbols to each graph of the generated member groups that indicate the position of the centroid vector corresponding to each member group. The group feature analysis unit 32 adjusts the position of each graph of the member groups so that the distance between the symbols indicating the positions of the centroid vectors is equal to the distance between the centroid vectors calculated in the processing of Sa10-1. The group feature analysis unit 32 reduces the dimensionality of each graph of the member groups generated in this way so that it can be displayed in two dimensions. The group feature analysis unit 32 outputs the graphs for each member group, which have been reduced to two dimensions, to the output device 3. As a result, for example, the graphs indicated by symbols 74, 75, and 76 corresponding to member groups 4, 5, and 6 shown in Figure 15 (hereinafter referred to as graph 74, graph 75, and graph 76, respectively) are displayed on the screen of the output device 3. Note that Figure 15 shows information for three member groups 4, 5, and 6 as an example, but all information for all member groups included in the member group data is displayed on the screen of the output device 3.

[0065] For example, in graphs 74, 75, and 76, each black circle represents a single response data, and the connecting lines between the black circles indicate a relationship between the response data. The diamond symbols indicated by the codes 74C, 75C, and 76C represent the positions of the centroid vectors of groups 4, 5, and 6, respectively (hereinafter referred to as centroid vector 74C, centroid vector 75C, and centroid vector 76C). The distances between centroid vector 74C and centroid vector 75C, between centroid vector 75C and centroid vector 76C, and between centroid vector 76C and centroid vector 74C are the distances between centroid vectors calculated in the Sa10-1 process.

[0066] The group feature analysis unit 32 refers to the contribution data generated in the processing of Sa10-2 and detects a predetermined number of words with a contribution level equal to or greater than a predetermined contribution threshold, starting with the words with the highest contribution, as common words between the groups. Here, the contribution threshold is predetermined to be, for example, "0.1", and the predetermined number is predetermined to be "3". In this case, for example, in the case of group 4 and group 5 shown in Figure 14, the contribution levels of "call", "outgoing call", and "external line" are all 0.1 or higher. Therefore, the group feature analysis unit 32 detects the words "call", "outgoing call", and "external line" as common words between group 4 and group 5, sorts the detected words in descending order of contribution, associates each sorted word with the others, and generates a common word table showing the corresponding contribution level. The group feature analysis unit 32 generates common word tables between group 5 and group 6, and between group 6 and group 4, by performing the same processing as that performed between group 4 and group 5.

[0067] The group feature analysis unit 32 refers to the contribution data generated in the processing of Sa10-2 and detects words that exist in only one of the affiliated groups as unique words. In the example shown in Figure 14, the word "one-way call" exists only in affiliated group 4 between affiliated group 4 and affiliated group 5, so its contribution is "0". Therefore, the group feature analysis unit 32 detects words with a contribution of "0", and if the contribution of the detected word is "0" between any of the affiliated groups, there is only one affiliated group that contains that word, and the word is treated as a unique word for that single affiliated group. Once the group feature analysis unit 32 detects unique words for each affiliated group, it generates a unique word table for each affiliated group that shows the unique words for that group.

[0068] The group feature analysis unit 32 displays each of the generated common word tables between the member groups and the group-specific word table on the screen of the output device 3, for example, as shown by reference numerals 61, 62, 63, 64, 65, and 66 in Figure 15 (Sa11), and then terminates the process. Reference numerals 61, 62, and 63 represent the common word tables between the member groups, and reference numerals 64, 65, and 66 represent the group-specific word tables.

[0069] (Effects of Document Data Processing Device 1) In the document data processing device 1, the similarity calculation unit 14 calculates the similarity between vectors of string data of the same item included in the response data. The grouping unit 15 selects response data that have direct and indirect co-relationships based on the similarity and the similarity threshold, and each of the selected combinations with co-relationships becomes a response data transition relationship consideration group. In this case, the response data to be processed by the document data processing device 1 is not document data that has been pre-classified into categories. The document data processing device 1 vectorizes each string data based on the words contained in each of the string data included in the response data, and classifies the response data into groups using the similarity between these vectors as an indicator. Therefore, by using the document data processing device 1, it is possible to determine the similarity between document data even if the document data has not been pre-classified into categories, and based on the determination result, similar document data can be efficiently extracted as groups.

[0070] Clustering, a machine learning technique, is commonly known as a method for grouping similar data. However, this method requires pre-determining how many classes to classify the data into. If the appropriate number of classes cannot be determined, dissimilar data may be classified into the same class, or similar data may be classified into different classes. Furthermore, data representing rare events may be misclassified as similar to other events or may be buried. While it is possible to predict the number of classes in advance when the amount of data is small, it is difficult to know how many classes exist in advance when dealing with large amounts of unorganized data. In contrast, the document data processing device 1 does not require pre-determining the number of classes and enables appropriate grouping of large amounts of unorganized correspondence data based on similarity.

[0071] The grouping unit 15 further divides and regroups the transition relationship consideration groups by detecting communities for each of the transition relationship consideration groups, thereby determining the group to which the response data belongs. As a result, it is possible to identify groups of response data with greater accuracy and precision. For example, even inexperienced respondents can efficiently determine how to answer a question by referring to the response data belonging to the group corresponding to the question. By referring to this accurate and precise grouping data, it is possible to know the number of response data belonging to each group, making it easy to extract frequently asked questions from a large amount of response data. Therefore, for example, the creation time can be significantly reduced compared to manually creating FAQs (Frequently Asked Questions) for response data.

[0072] In primary contact centers, it is necessary to respond quickly to a large volume of inquiries, and many inquiries pertain to the same issue. Furthermore, because primary contact centers are only first-line, they do not track down inquiries until they are resolved, and therefore, past response data does not always contain the appropriate answer to the question. Consequently, simply searching through the accumulated past response data in contact centers often yields results that include multiple similar incidents, and finding the desired response data with the appropriate answer from among them can be very time-consuming.

[0073] In such cases, it is conceivable to build a system that uses AI (Artificial Intelligence) or machine learning to train a large amount of response data as training data, making it easier to find the desired response data. However, in AI (Artificial Intelligence) and machine learning, the quality of the training data is directly related to the accuracy rate. Therefore, it is necessary to cleanse the response data to improve its quality. In this case, the group representative data generation unit 31 in the document data processing device 1 generates group representative response data that represents each group from the response data belonging to each group of response data grouped by the grouping unit 15. By using this group representative response data, as explained with reference to Figures 11 to 13, the response data can be compressed, i.e., cleansed, thereby improving the quality of the response data, and making it possible to use high-quality response data as training data for AI and machine learning.

[0074] The group feature analysis unit 32 of the document data processing device 1 calculates the centroid vector of each member group and, based on the calculated centroid vectors, calculates the distance between member groups and the contribution of common words between the member groups. This allows the analysis results, such as those explained with reference to Figure 15, to be output to the output device 3, visualizing the characteristics of the member groups and enabling understanding, for example, why a particular grouping was performed.

[0075] <Second Embodiment> Hereinafter, an embodiment relating to this disclosure will be described with reference to the figures. Contact centers and the like operate on a daily basis, and if a period of time has passed since the grouping of response data was performed by the document data processing device 1 shown in Figure 1, new response data will be accumulated in the response data storage unit 2. In order to reflect the contents of the newly accumulated response data, it is necessary to update the transition relationship-considering group data and the affiliated group data. The document data processing device 1a shown in Figure 16 is a device used in such update situations.

[0076] As shown in Figure 16, the document data processing device 1a has a configuration in which the similarity calculation unit 14 is replaced with a similarity calculation unit 14a in the document data processing device 1 shown in Figure 1, and further includes a group combination adjustment unit 19. Although the acquisition unit 11, word extraction unit 12, vectorization unit 13, high-similarity data selection unit 16, and transition relationship-considering group determination unit 17 are given the same reference numerals in both the document data processing device 1 and the document data processing device 1a, there are differences in the processing targets in each device, namely, corresponding to the differences in the processing targets, namely, response data and incremental response data which are added response data. In addition, although not shown in Figure 16, the document data processing device 1a also includes a group representative data generation unit 31 and a group feature analysis unit 32, which are also present in the document data processing device 1.

[0077] The similarity calculation unit 14a has the configuration of the similarity calculation unit 14, and further has the following configuration. Based on the incremental vector data generated by the vectorization unit 13 from the incremental response data, the similarity calculation unit 14a calculates the similarity between the incremental response data for all combinations of two incremental response data selected from a plurality of incremental response data. Based on the existing vector data and the incremental vector data, the similarity calculation unit 14a calculates the similarity between the existing response data and the incremental response data for all combinations of existing response data and incremental response data selected from the existing response data and the incremental response data. The existing vector data is the vector data generated by the vectorization unit 13 in the process of Sa3 in Figure 5 from the existing response data (hereinafter referred to as existing response data), which is the response data before the incremental response data is added.

[0078] The group combination adjustment unit 19 detects, for each incremental response data belonging to the incremental transition relationship consideration group, the number of combinations between the incremental response data and the existing transition relationship consideration group formed by the existing response data, and the number of combinations within the incremental transition relationship consideration group, which is indicated by the number of other incremental response data to which the incremental response data is combined. The existing transition relationship consideration group is the transition relationship consideration group determined by the transition relationship consideration group determination unit 17 in the process of Sa6 in Figure 5. The incremental transition relationship consideration group is the transition relationship consideration group determined by the transition relationship consideration group determination unit 17 based on the similarity calculated by the similarity calculation unit 14a in response to the addition of incremental response data and the similarity threshold. The group combination adjustment unit 19 selects the destination to which each of the incremental response data belongs based on the number of combinations to be detected, the similarity calculated by the similarity calculation unit 14a in response to the addition of incremental response data, and the transition relationship consideration groups already formed from the existing response data. In accordance with this selection, it regroups the transition relationship consideration groups already formed from the existing response data and the incremental transition relationship consideration groups.

[0079] (Regrouping process when adding response data by document data processing device 1a) The following describes the regrouping process performed by the document data processing device 1a when adding response data, with reference to the flowchart shown in Figure 17. Before the flowchart shown in Figure 17 is started, the state of the intermediate data storage unit 20 is assumed to be that it stores at least the existing vector data generated in the process of Sa3 in Figure 5 and the transition relationship-considering group data (hereinafter referred to as existing transition relationship-considering group data) generated in the process of Sa6.

[0080] When response data is added to and recorded in the response data storage unit 2, the acquisition unit 11 acquires the added response data one by one as incremental response data and outputs the acquired incremental response data to the word extraction unit 12. When the acquisition unit 11 has acquired all the incremental response data stored in the response data storage unit 2 and finished outputting to the word extraction unit 12, it outputs an incremental completion notification signal to the word extraction unit 12 (Sb1).

[0081] Each time the word extraction unit 12 takes in incremental response data output by the acquisition unit 11, it performs the same process as the word extraction process in the Sa2 process in Figure 5 on the acquired incremental response data to generate word data. The word extraction unit 12 records the generated word data as incremental word data in the intermediate data storage unit 20. For example, the incremental response data output by the acquisition unit 11 includes discrimination information that allows it to be determined that it is added response data. When the word extraction unit 12 generates word data for response data that includes such discrimination information, it records the generated word data in the intermediate data storage unit 20 as incremental word data that can be distinguished from existing word data. The word extraction unit 12 receives an incremental completion notification signal from the acquisition unit 11, and furthermore, when it has completed the generation and recording of incremental word data corresponding to all the incremental response data acquired up to the time of receiving the incremental completion notification signal, it outputs the incremental completion notification signal to the vectorization unit 13 (Sb2).

[0082] When the vectorization unit 13 receives an increment completion notification signal from the word extraction unit 12, it performs the same process as the vectorization process in the Sa3 process of Figure 5 on all the incremental word data stored in the intermediate data storage unit 20 to generate vector data. The vectorization unit 13 records the generated vector data in the intermediate data storage unit 20 as incremental vector data that can be distinguished from existing vector data. When the vectorization unit 13 has finished generating and recording incremental vector data corresponding to all the incremental word data stored in the intermediate data storage unit 20, it outputs an increment completion notification signal to the similarity calculation unit 14a (Sb3).

[0083] When the similarity calculation unit 14a receives an increment completion notification signal from the vectorization unit 13, the intermediate data storage unit 20 stores the increment vector data and the existing vector data. The similarity calculation unit 14a can also employ a procedure in which it calculates the similarity between all response data, including the existing response data and the increment response data, and then re-determines the transition relationship consideration group based on the calculated similarity. However, if the total number of response data, including the increment response data, increases, the amount of calculation required for similarity also increases.

[0084] To suppress this increase in computational load, the similarity calculation unit 14a focuses on calculating the similarity necessary for re-determining the transition relationship consideration group due to the addition of incremental response data. Specifically, the similarity between N existing response data is excluded from the calculation target of the similarity calculation unit 14a. The similarity between N existing response data is the similarity already calculated in the process of Sa4 in Figure 5, and the existing transition relationship consideration group data has already been generated and stored in the intermediate data storage unit 20 based on this similarity. Therefore, even if the similarity between N existing response data is excluded from the calculation target of the similarity calculation unit 14a, the similarity between N existing response data necessary for re-grouping can be obtained from the existing transition relationship consideration group data.

[0085] For example, if there are N existing response data entries, and 5 incremental response data entries are added, the total number of combinations of response data entries including the existing and incremental response data is:N From C2 (N+5) It increases to C2. The number of combinations for the increase is (N+5) C2- N C2 = 5N + 10. Of these "5N + 10" combinations, the "5N" portion corresponds to the combinations of existing response data and incremental response data shown in the upper right white grid area of ​​the relationship between response data in Figure 18. The "10" portion corresponds to the combinations of incremental response data shown in the lower right white grid area.

[0086] When the similarity calculation unit 14a receives an increment completion notification signal from the vectorization unit 13, it outputs an increment start notification signal to the high-similarity data selection unit 16. The similarity calculation unit 14a selects a combination of two incremental vector data from the incremental vector data stored in the intermediate data storage unit 20 that corresponds to the combination of incremental response data shown in the white grid area in the lower right of Figure 18. The similarity calculation unit 14a calculates the similarity between the incremental response data for the two selected incremental vector data using the same process as the process for calculating similarity in the Sa4 process. That is, the similarity calculation unit 14a calculates the similarity between the question text vectors and the similarity between the answer text vectors contained in each of the two selected incremental vector data. The similarity calculation unit 14a generates similarity data that includes the case ID of each of the two selected incremental vector data, the similarity between the question text vectors, and the similarity between the answer text vectors, and outputs the generated similarity data to the high-similarity data selection unit 16. The similarity calculation unit 14a generates similarity data for all combinations of the two incremental vector data and outputs it to the high-similarity data selection unit 16 (Sb4).

[0087] Once processing Sb4 is complete, the similarity calculation unit 14a selects a combination of existing vector data and incremental vector data from the existing vector data and incremental vector data stored in the intermediate data storage unit 20 that corresponds to the portion of the combination of existing response data and incremental response data shown in the white grid area in the upper right of Figure 18. The similarity calculation unit 14a calculates the similarity between the existing response data and the incremental response data for the selected combination of existing vector data and incremental vector data using the same process as the process for calculating similarity in processing Sa4. That is, the similarity calculation unit 14a calculates the similarity between the question sentence vectors and the answer sentence vectors contained in the existing vector data and incremental vector data, respectively.

[0088] The similarity calculation unit 14a generates similarity data that includes the case IDs of the selected existing vector data and incremental vector data, the similarity between question vectors, and the similarity between answer vectors, and outputs the generated similarity data to the high-similarity data selection unit 16. When the similarity calculation unit 14a has generated similarity data for all combinations of existing vector data and incremental vector data and output it to the high-similarity data selection unit 16, it outputs an incremental completion notification signal to the high-similarity data selection unit 16 (Sb5). Note that the order of processing Sb4 and Sb5 may be reversed. If reversed, the process in which the similarity calculation unit 14a outputs an incremental completion notification signal to the high-similarity data selection unit 16, which is performed at the end of processing Sb5, will be performed at the end of processing Sb4.

[0089] When the high-similarity data selection unit 16 receives an increment start notification signal from the similarity calculation unit 14a, it outputs the increment start notification signal to the transition relationship-considering group determination unit 17. The high-similarity data selection unit 16 takes in the similarity data output by the similarity calculation unit 14a. For each of the similarity data taken in between receiving the increment start notification signal from the similarity calculation unit 14a and receiving the increment completion notification signal, the high-similarity data selection unit 16 performs the same process as the process of selecting similarity data based on the similarity threshold in the process of Sa5 in Figure 5. The high-similarity data selection unit 16 outputs the selected similarity data to the transition relationship-considering group determination unit 17. When the process of selecting each of the similarity data taken in between receiving the increment start notification signal and receiving the increment completion notification signal based on the similarity threshold is completed, the high-similarity data selection unit 16 outputs an increment completion notification signal to the transition relationship-considering group determination unit 17 (Sb6).

[0090] The transition relationship-considering group determination unit 17 receives an increment start notification signal from the high-similarity data selection unit 16 and then takes in the similarity data output by the high-similarity data selection unit 16. The transition relationship-considering group determination unit 17 generates transition relationship-considering group data by performing the same process as the process for generating transition relationship-considering group data in the process of Sa6 in Figure 5 on the similarity data taken in from the time it receives the increment start notification signal to the time it receives the increment completion notification signal from the high-similarity data selection unit 16. The transition relationship-considering group determination unit 17 records the generated transition relationship-considering group data as incremental transition relationship-considering group data in the intermediate data storage unit 20 and outputs a completion notification signal to the group combination adjustment unit 19 (Sb7).

[0091] When the group combination adjustment unit 19 receives a completion notification signal from the transition relationship consideration group determination unit 17, it adjusts the combination relationship between the incremental response data and the existing response data based on the existing transition relationship consideration group data and the incremental transition relationship consideration group data stored in the intermediate data storage unit 20.

[0092] As for the coupling relationship between the incremental response data and the existing transition relationship consideration group, six patterns of coupling relationships are assumed, as shown in Figure 19. The group coupling adjustment unit 19 determines which of the six coupling relationship patterns the coupling relationship between the incremental response data and the existing transition relationship consideration group corresponds to, and performs individual coupling adjustments for each determined pattern.

[0093] Figure 20 shows an example of the coupling relationship between incremental response data and existing transition relationship consideration groups. In Figure 20, black circles represent existing response data, white circles represent incremental response data, and solid or dashed connecting lines indicate a coupling relationship. Existing groups A, B, C, and D are existing transition relationship consideration groups and are identified from the existing transition relationship consideration group data.

[0094] The incremental transition relationship consideration groups identified from the incremental transition relationship consideration group data are formed between incremental response data that have a coupling relationship, or between incremental response data and existing response data. Therefore, in the example shown in Figure 20, one incremental transition relationship consideration group is formed by the incremental response data labeled 81 to 85 and the existing response data labeled 91 to 94, 97, and 98; another incremental transition relationship consideration group is formed by the incremental response data labeled 86 and the existing response data labeled 96; and yet another incremental transition relationship consideration group is formed by the incremental response data labeled 87 and the existing response data labeled 95 and 99. Hereafter, the incremental response data labeled 81 will be referred to as incremental response data 81, and the incremental response data labeled 82 to 87 and the existing response data labeled 91 to 99 will be referred to similarly.

[0095] The group combination adjustment unit 19 detects, for each incremental response data belonging to the incremental transition relationship consideration group, the number of connections between the incremental response data and the existing transition relationship consideration group, and the number of connections within the incremental transition relationship consideration group, which is indicated by the number of other incremental response data to which the incremental response data is connected. For example, in the case of incremental response data 82, since incremental response data 82 is connected with existing response data 92, 93, and 94 belonging to existing group B, the group combination adjustment unit 19 detects "3" as the number of connections between incremental response data 82 and existing group B. Also, since incremental response data 82 is connected with other incremental response data 81 and 84, the group combination adjustment unit 19 detects "2" as the number of connections within the incremental transition relationship consideration group for incremental response data 82.

[0096] In the case of (Pattern 1), the incremental response data 86 is applicable. The incremental response data 86 is only linked with existing response data 96 which belongs to existing group C. Therefore, the group linking adjustment unit 19 sets the group to which the incremental response data 86 belongs to existing group C.

[0097] For patterns 2, 4, and 5, if we consider the existing transition relationship consideration group and the incremental transition relationship consideration group as the same group, we can consider that one incremental response data belongs to multiple groups. In this case, the group to which the one incremental response data belongs is the group with the most connections, and the connections with other groups are removed. If the number of connections between multiple groups is the same, the group with the greater similarity is chosen to which the one incremental response data belongs, and the connections with other groups are removed.

[0098] The incremental response data 87 corresponds to (Pattern 2). The incremental response data 87 is coupled with existing response data 95 belonging to existing group B and existing response data 99 belonging to existing group D. In other words, the incremental response data 87 has one coupling with existing group B and one with existing group D. In this case, the group coupling adjustment unit 19 refers to the incremental transition relationship-considering group data and refers to the similarity between the incremental response data 87 and the existing response data 95, and the similarity between the incremental response data 87 and the existing response data 99, and determines which group the incremental response data 87 belongs to based on the greater similarity.

[0099] Note that there are two types of similarity scores: one for the question and one for the answer. For example, the option with the higher similarity score between both options should be chosen as the affiliation. If one option has a high similarity score, but the other is the same or lower, the option with the higher sum of both similarity scores may be chosen as the affiliation, or the option with the higher average of both similarity scores may be chosen as the affiliation.

[0100] In this case, since the similarity between the incremental response data 87 and the existing response data 99 is greater than the similarity between the incremental response data 87 and the existing response data 95, the group combination adjustment unit 19 assigns the incremental response data 87 to group D.

[0101] Furthermore, if the number of combinations and the similarity are the same, the affiliation may be determined as follows, for example, in (a), (b), (c), and (d) below. For example, suppose that the case IDs of all response data, including incremental response data, are recorded in the response data storage unit 2 in the order in which they are recorded, with the largest number being the largest. In other words, the response data with a smaller case ID number is older response data. In this case, the affiliation may be determined as follows, in (a) and (b).

[0102] (i) The customer service data with the smaller case ID number among the candidate customer service data may be used as the customer service location.

[0103] (b) If the phenomenon is observed in which the same problem tends to occur at a certain time, the case ID number of the incremental response data for which a department needs to be determined will be compared with the case ID number of the candidate response data for the department, and the response data with the closer number, i.e., the one with the smaller absolute difference in the case ID numbers, will be designated as the department.

[0104] It should be assumed that all response data, including incremental response data, is assigned a registration date and time and recorded in the response data storage unit 2. In this case, instead of method (a), the response data with the earlier registration date and time, i.e., the response data registered in the past, may be used as the affiliation. Instead of method (b), the response data that is a candidate for the affiliation and has a registration date and time close to the registration date and time assigned to the incremental response data for which the affiliation needs to be determined may be used as the affiliation.

[0105] (h) In the example described above, the word extraction unit 12 excludes words of parts of speech other than nouns when generating word data and incremental word data. In contrast, the word extraction unit 12 generates morphological analysis result data for each string data, which includes all words of parts of speech extracted from the string data by morphological analysis in the order they were extracted, separately from the word data and incremental word data, and records it in the intermediate data storage unit 20. In this case, the words included in the morphological analysis result data of the incremental response data for which the destination needs to be determined are compared with the words included in the morphological analysis result data of each candidate response data for the destination, and the candidate response data for the destination with the larger number of matching words is designated as the destination.

[0106] (ii) The identity of word order, in other words, the context may be taken into consideration. For example, suppose in the incremental word data of incremental response data for which an affiliation needs to be determined, the words are arranged in the order (word A, word B, word C), in the word data of one candidate response data for affiliation, the words are arranged in the order (word A, word C, word B), and in the word data of the other candidate response data for affiliation, the words are arranged in the order (word C, word A, word B). In this case, the identity of word order is compared by determining how many times adjacent words must be swapped to match the order of (word A, word B, word C) in the incremental response data for which an affiliation needs to be determined. For one of the (word A, word C, word B) cases, swapping the order of word C and word B will result in the order (word A, word B, word C). In contrast, for the other (word C, word A, word B), the order of word C and word A is swapped to become (word A, word C, word B), and then the order of word C and word B is swapped again to become (word A, word B, word C). In other words, one instance can be matched to the word order of the incremental response data whose affiliation needs to be determined with just one swap, whereas the other instance requires two swaps. Therefore, in this case, the response data candidate for the affiliation of one instance, which has the order (word A, word C, word B) with fewer swaps, has a higher degree of identity and is closer to the context of the incremental response data whose affiliation needs to be determined, so the response data candidate for the affiliation of one instance is selected as the affiliation. Alternatively, instead of using the incremental word data of the incremental response data and the word data of the response data as described above, the morphological analysis result data explained in (c) may be stored in the intermediate data storage unit 20, and an identity comparison may be performed based on the words included in the morphological analysis result data.

[0107] Furthermore, the methods described in (a), (b), (c), and (d) above may be combined to determine the affiliation.

[0108] The incremental response data 81 and 82 correspond to (Pattern 4). Incremental response data 81 is linked to existing response data 91, which belongs to existing group A, and to incremental response data 82, 83, and 84. The number of links between incremental response data 81 and existing group A is "1". In contrast, the number of links within the incremental transition relationship consideration group for incremental response data 81 is "3". Therefore, the group linking adjustment unit 19 sets the destination of incremental response data 81 to the incremental transition relationship consideration group to which incremental response data 81 belongs.

[0109] The incremental response data 82 is linked to the existing response data 92, 93, and 94 belonging to existing group B, and to the incremental response data 81 and 84. The number of links between incremental response data 82 and existing group B is "3". In contrast, the number of links within the incremental transition relationship-considering group of incremental response data 82 is "2". Therefore, the group linking adjustment unit 19 sets the group to which incremental response data 82 belongs to existing group B.

[0110] The incremental response data 85 corresponds to (Pattern 5). The incremental response data 85 is combined with existing response data 97 belonging to existing group C, existing response data 98 ​​belonging to existing group D, and incremental response data 84. The number of combinations between incremental response data 85 and existing group C is "1". The number of combinations between incremental response data 85 and existing group D is "1". The number of combinations within the incremental transition relationship-considering group of incremental response data 84 is "1". Therefore, since the number of combinations is the same at "1", the group combination adjustment unit 19 determines the group to which incremental response data 85 belongs according to the similarity of each combination. Here, since the similarity between incremental response data 85 and existing response data 98 ​​is the greatest, the group combination adjustment unit 19 assigns the group to which incremental response data 85 belongs.

[0111] The incremental response data 83 and 84 correspond to (Pattern 6). Since the incremental response data 83 and 84 are not linked with the existing response data, the group linking adjustment unit 19 sets the group to which the incremental response data 83 and 84 belong to the incremental transition relationship consideration group to which the incremental response data 83 and 84 belong.

[0112] Incremental response data corresponding to (Pattern 3) refers to incremental response data that is not combined with any other incremental response data and is not combined with any existing response data. In other words, its similarity is below the similarity threshold. Since such incremental response data is not selected by the high-similarity data selection unit 16, it is not subject to adjustment by the group combination adjustment unit 19.

[0113] Therefore, when the group coupling adjustment unit 19 performs coupling adjustment on the example shown in Figure 20, the incremental response data 81 to 87 will have the coupling relationship shown in Figure 21. Each of the existing groups A, B, C, D and the incremental group shown in Figure 21 becomes a new transition relationship consideration group. The group coupling adjustment unit 19 generates new transition relationship consideration group data for the new transition relationship consideration group, records the generated new transition relationship consideration group data in the intermediate data storage unit 20, and outputs a completion notification signal to the community detection unit 18 (Sb8).

[0114] When the community detection unit 18 receives a completion notification signal from the group combination adjustment unit 19, it reads the new transition relationship-considering group data stored in the intermediate data storage unit 20 and performs the same processing as the processing in Sa7 of Figure 5 on the read new transition relationship-considering group data (Sb9). As a result, the intermediate data storage unit 20 stores new belonging group data corresponding to the new transition relationship-considering group data. Subsequently, the group representative data generation unit 31 and the group feature analysis unit 32 perform the processing in Sa8, Sa9, Sa10-1, Sa10-2, and Sa11 of Figure 5 on the new belonging group data stored in the intermediate data storage unit 20 as group representative response data generation and group feature analysis processing (Sb10), and the processing is completed.

[0115] (Effects of the document data processing device 1a) While it is possible to group existing response data and incremental response data together using the document data processing device 1, such grouping would make it impossible to understand the relationships with transitional relationship groups and affiliated groups formed by existing response data, and the grouping process would take a long time. In contrast, by using the document data processing device 1a shown in Figure 16, the similarity calculation unit 14a calculates the similarity of question sentences and answer sentences between incremental response data based on the vector corresponding to each of the incremental response data, and calculates the similarity of question sentences and answer sentences between incremental response data and existing response data based on the vector corresponding to each of the incremental response data and the vector corresponding to each of the existing response data.

[0116] The transition relationship consideration group determination unit 17 focuses on existing response data that have a high similarity relationship with the incremental response data and determines the incremental transition relationship consideration group, which represents the connection relationships between the incremental response data and the connection relationships between the incremental response data and the existing response data. The group combination adjustment unit 19 determines the destination of the incremental response data based on the number of connections between the incremental response data and the existing transition relationship consideration group, the number of connections within the incremental transition relationship consideration group indicated by the number of other incremental response data that the incremental response data connects to, and the similarity, and regroups the existing transition relationship consideration group and the incremental transition relationship consideration group. In this way, by utilizing the existing transition relationship consideration group and forming a new transition relationship consideration group through regrouping, it becomes possible to grasp the relationship between the two groups. Furthermore, as explained with reference to Figure 18, since the similarity between existing response data is excluded from the similarity calculation target in the similarity calculation unit 14a, the amount of computation can be reduced, and the time required for grouping processing can be reduced.

[0117] (Other configuration examples common to document data processing devices 1 and 1a) The following describes other configuration examples common to the document data processing device 1 shown in Figure 1 and the document data processing device 1a shown in Figure 16.

[0118] The vectorization unit 13 may perform vectorization by assigning weights to specific words that are predetermined when performing vectorization. For example, the vectorization unit 13 may assign small weights to unimportant words and large weights to important words in the grouping performed by the transitional relationship-considering group determination unit 17 and the community detection unit 18. The analysis results shown in Figure 15 may be used as an indicator for determining the importance of each word in the grouping performed by the transitional relationship-considering group determination unit 17 and the community detection unit 18. For example, on the first run, the process shown in Figure 5 is performed without assigning weights to the words in the vectorization unit 13, and the analysis results shown in Figure 15 are displayed on the screen of the output device 3. The words shown in the common word tables 61 to 63 displayed on the screen of the output device 3 are selected as important words, and the words shown in the specific word tables 64 to 65 are selected as unimportant words, and weight values ​​are determined for the selected words. Then, the process shown in Figure 5 may be performed again, or when incremental response data is added to the response data storage unit 2, the process shown in Figure 17 may be performed to regroup and determine the group to which the data belongs. After performing the process shown in Figure 17, the screen of the output device 3 may be referred to again, important and unimportant words may be selected again, weight values ​​for the selected words may be determined, and the process shown in Figure 5 may be performed again. This makes it possible to identify groups with greater accuracy and precision.

[0119] In the document data processing devices 1 and 1a, the group representative data generation unit 31 calculates a centroid vector for each group to which it belongs and generates group representative response data. Alternatively, the group representative data generation unit 31 may select one response data that represents the group to which it belongs using an indicator other than the centroid vector and generate group representative response data. For example, the group representative data generation unit 31 may have the contents of all response data belonging to the group to be summarized by an LLM (Large Language Model) and generate group representative response data. The group representative data generation unit 31 may generate group representative response data by concatenating the contents of all the "Question" items of all the response data belonging to the group to form the "Question" item of the group representative response data, concatenating the contents of all the "Answer" items of all the response data to form the "Answer" item of the group representative response data, and concatenating the contents of all the "Other" items of all the response data to form the "Other" item of the group representative response data.

[0120] As explained with reference to Figure 5, if the group representative data generation unit 31 calculates the centroid vector in the processing of Sa8, the group feature analysis unit 32 may perform the processing of Sa10-1 and Sa10-2 by reusing the centroid vector calculated by the group representative data generation unit 31 without performing the processing of Sa9.

[0121] In the document data processing devices 1 and 1a, the group feature analysis unit 32 detects common words shared among member groups and unique words specific to each member group based on their contribution. Alternatively, the group feature analysis unit 32 may analyze and output various statistical information about the member groups, such as detecting words that differ significantly among member groups or words common to all member groups, based on their contribution. Furthermore, the group feature analysis unit 32 may classify the words present in the member groups using IDF (Inverse Document Frequency) vectors. For example, when the group feature analysis unit 32 calculates the centroid vector of a member group, it calculates the IDF vector for each word in that group. The group feature analysis unit 32 may classify words with large calculated IDF values ​​as words present in fewer member groups, and words with small IDF values ​​as words present in more member groups, and output the classification results as analysis results to the output device 3.

[0122] In the document data processing device 1,1a, the group feature analysis unit 32 performs analysis processing on the affiliated group based on the affiliated group data, i.e., the processing of Sa9, Sa10-1, Sa10-2, and Sa11 in Figure 5. However, similar analysis processing may be performed on groups considering transition relationships.

[0123] In the document data processing devices 1 and 1a, the community detection unit 18 generates group data in the data format shown in Figure 10. However, it may also generate group data that includes the fields "Case ID" and "Group Affiliation," excluding the fields "Question," "Answer," and "Other" from the data format shown in Figure 10. Even with group data in such a format, it is possible to identify which group identification information corresponds to the response data corresponding to the Case ID. However, in this case, the group data will not include the fields "Question," "Answer," and "Other." Therefore, when the group representative data generation unit 31 generates group representative response data by referring to the group data, it will obtain the response data corresponding to the Case ID shown in the "Case ID" field from the response data storage unit 2 and generate the group representative response data.

[0124] In the document data processing devices 1 and 1a, the community detection unit 18 may generate the group data including information showing the connection relationships between response data associated with each case ID included in the group data. In this case, the group feature analysis unit 32 can generate graphs 74, 75, and 76 of the response data belonging to each of the groups shown in Figure 15 using the connection relationships between the response data included in the group data, without referring to the group data considering transition relationships.

[0125] The document data processing device 1,1a may be configured without a community detection unit 18. In this case, the transitional relationship-considering group becomes the final group of the response data. The group representative data generation unit 31 and the group feature analysis unit 32 will perform the processing that was previously performed on the belonging group data on the transitional relationship-considering group data. Alternatively, the document data processing device 1,1a may be configured to include a community detection unit 18, but the user may be able to choose whether or not to have the community detection unit 18 perform processing.

[0126] In the document data processing devices 1 and 1a, the high-similarity data selection unit 16 selects two combinations of response data as having a high similarity relationship if the similarity of the question sentences corresponding to the combination of two response data is equal to or greater than a similarity threshold, and the similarity of the answer sentences corresponding to the combination of two response data is also equal to or greater than the similarity threshold. Alternatively, the high-similarity data selection unit 16 may select two combinations of response data as having a high similarity relationship if the sum of the similarity of the similarity data of the question sentences and the similarity of the corresponding answer sentences in the combination of two response data is equal to or greater than a predetermined similarity threshold.

[0127] In the document data processing device 1,1a, the high-similarity data selection unit 16 determines whether the similarity threshold is greater than or equal to the similarity threshold and selects correspondence data or incremental correspondence data with a similarity of greater than or equal to the similarity threshold as high-similarity data. Alternatively, depending on how the similarity threshold is defined, the high-similarity data selection unit 16 may determine whether the similarity threshold is exceeded and select correspondence data or incremental correspondence data with a similarity exceeding the similarity threshold as high-similarity data.

[0128] Similarly, the group feature analysis unit 32 determines whether the contribution is above the contribution threshold and selects words with a contribution above the contribution threshold. Alternatively, depending on how the contribution threshold is defined, the group feature analysis unit 32 may determine whether the contribution exceeds the contribution threshold and select words with a contribution exceeding the contribution threshold.

[0129] The document data processing device 1,1a is configured to process both the string data of the question sentence indicated in the "Question" item and the string data of the answer sentence indicated in the "Answer" item included in the response data. Alternatively, the document data processing device 1,1a may perform processing on either the string data of the question sentence or the string data of the answer sentence.

[0130] <Third Embodiment> An embodiment of this disclosure will be described below with reference to a diagram. In a contact center or similar setting, if a respondent who is assisting a caller wants to provide an answer by referring to past response data, they will search the response data. In this case, if the respondent cannot come up with an appropriate search term to obtain the desired response data, the search accuracy will decrease, and they will be unable to provide an appropriate answer to the caller. In contrast, it is desirable to improve the probability of obtaining the desired response data without reducing search accuracy, even if the search term thought up by the respondent does not contain enough information to obtain the desired response data, thereby enabling them to provide a more appropriate answer to the caller.

[0131] The document data processing device 1b shown in Figure 22 is a modified version of the document data processing device 1 shown in Figure 1, with the acquisition unit 11 replaced by the acquisition unit 11a, the word extraction unit 12 replaced by the word extraction unit 12a, and further comprising an occurrence word count unit 41, a co-occurrence word data generation unit 42, a search unit 51, and a co-occurrence word recommendation unit 54. Although not shown in Figure 22, the document data processing device 1b may also include a vectorization unit 13, similarity calculation units 14, 14a, a grouping unit 15, a group combination adjustment unit 19, a group representative data generation unit 31, and a group feature analysis unit 32, which are also present in the document data processing devices 1 and 1b.

[0132] The acquisition unit 11a is connected to the response data storage unit 2. When it receives a response data acquisition request signal from another functional unit, it reads response data one by one from the response data storage unit 2 and outputs the read response data to the requesting functional unit. Once the acquisition unit 11a has acquired all the response data stored in the response data storage unit 2 and finished outputting to the requesting functional unit, it outputs a completion notification signal to the requesting functional unit. The acquisition unit 11a may also have the same configuration as the acquisition unit 11.

[0133] When the word extraction unit 12a receives a word extraction request signal containing string data from another functional unit, it extracts words from the string data contained in the received word extraction request signal using the same procedure as the word extraction unit 12 performs, and outputs a word extraction completion signal containing a list of the extracted words to the requesting functional unit. Note that the word extraction unit 12a may have the same configuration as the word extraction unit 12.

[0134] The word occurrence counting unit 41 selects each word included in a word group containing one or more words as a reference word. Here, the word group is a word group of a predetermined item included in the word data stored in the intermediate data storage unit 20, or a word group included in the word extraction completion signal output by the word extraction unit 12a. For each word group from which a reference word is selected, the word occurrence counting unit 41 considers words that exist within a predetermined number of words before and after the reference word as co-occurring words of the reference word, and counts the number of times the co-occurring words appear within that predetermined number of words before and after the reference word.

[0135] The co-occurrence word data generation unit 42 calculates the total number of occurrences counted by the occurrence word count unit 41 for each word group for combinations of a reference word and a co-occurring word corresponding to that reference word. The co-occurrence word data generation unit 42 generates co-occurrence word data that associates each of the calculated total values ​​with the corresponding combination of a reference word and a co-occurring word.

[0136] The search unit 51 is connected to the output device 3 and the input device 4, and comprises a search processing unit 53 and a search query acquisition unit 52. Here, the input device 4 is, for example, a keyboard, mouse, or touch panel. When the search processing unit 53 receives a search request signal from the input device 4, it detects response data from the response data storage unit 2 that includes the string data indicated by the search query in a format that conforms to the search conditions, based on the search conditions and search query contained in the received search request signal. The search processing unit 53 outputs the detected response data to the output device 3.

[0137] Here, the search condition is, for example, either an AND search condition or an OR search condition. When the search query contains multiple string data separated by spaces and the condition is an AND search, the search processing unit 53 detects corresponding data that includes all of the multiple string data indicated by the search query in any part of the string data of a predetermined item. Conversely, when the search query contains multiple string data and the condition is an OR search, the search processing unit 53 detects corresponding data that includes at least one of the multiple string data indicated by the search query in any part of the string data of a predetermined item.

[0138] The search query acquisition unit 52 displays a search query input screen on the output device 3 for writing search queries. When a string is written into the search query input box on the displayed search query input screen, the search query acquisition unit 52 captures the written string as string data and outputs the string data as a search query to the co-occurring word recommendation unit 54.

[0139] The co-occurring word recommendation unit 54 detects co-occurring word data from the co-occurring word data generated by the co-occurring word data generation unit 42, using each word indicated by the search query output by the search query acquisition unit 52 as a base word. If the co-occurring word data to be detected contains a common co-occurring word, the co-occurring word recommendation unit 54 takes the sum of the occurrence counts of the common co-occurring word as the occurrence count of the common co-occurring word. The co-occurring word recommendation unit 54 sorts each of the co-occurring words included in the detected co-occurring word data in descending order of their corresponding occurrence counts, and outputs the co-occurring words up to the top predetermined position to the search query acquisition unit 52.

[0140] The intermediate data storage unit 20 stores at least co-occurring word data. If the document data processing device 1b comprises a vectorization unit 13, a similarity calculation unit 14, a grouping unit 15, and the word extraction unit 12a further comprises the configuration of the word extraction unit 12, the intermediate data storage unit 20 stores word data, vector data, similarity data, transition relationship-considering group data, and belonging group data.

[0141] (Generation of co-occurring word data by document data processing device 1b) The following describes the process by which the document data processing device 1b generates co-occurring word data, referring to the flowchart shown in Figure 23. Here, we will describe an example where the "question" item of the response data is predetermined as one of the predetermined items mentioned above.

[0142] The word occurrence count unit 41 determines whether or not word data is stored in the intermediate data storage unit 20. If the word occurrence count unit 41 determines that word data is stored in the intermediate data storage unit 20 (Sc1, Yes), it retrieves the group of question words contained in each of the word data stored in the intermediate data storage unit 20 (Sc2).

[0143] On the other hand, if the word counting unit 41 determines that no word data is stored in the intermediate data storage unit 20 (Sc1, No), it outputs a response data acquisition request signal to the acquisition unit 11a. When the acquisition unit 11a receives the response data acquisition request signal from the word counting unit 41, it reads the response data one by one from the response data storage unit 2 and outputs the read response data to the word counting unit 41. When the acquisition unit 11a has acquired all the response data stored in the response data storage unit 2 and finished outputting to the word counting unit 41, it outputs a completion notification signal to the word counting unit 41.

[0144] Each time the word count unit 41 takes in response data output by the acquisition unit 11a, it outputs a word extraction request signal to the word extraction unit 12a that includes the string data of the question sentence indicated in the "Question" item of the acquired response data. When the word extraction unit 12a receives the word extraction request signal from the word count unit 41, it extracts words from the string data included in the received word extraction request signal and outputs a word extraction completion signal to the word count unit 41 that includes a group of words listing the extracted words. The word count unit 41 takes in the word extraction completion signal output by the word extraction unit 12a and uses the group of words included in the acquired word extraction completion signal as the word group of the question sentence. The word count unit 41 repeats the above procedure for each piece of response data until it receives a completion notification signal from the acquisition unit 11a, thereby acquiring the word group of the question sentence corresponding to each of the string data of the question sentences of all the response data stored in the response data storage unit 2 (Sc3).

[0145] The word sets of question sentences obtained by the word counting unit 41 through the processing of Sc2 and Sc3 are all the same, as they are the same word sets extracted by the word extraction units 12 and 12a from each of the string data of question sentences corresponding to each predetermined item of the response data stored in the response data storage unit 2, namely the "question" item.

[0146] The word counting unit 41 selects one word set from the word sets of question sentences obtained by processing Sc2 or Sc3, and selects each word in the selected word set of question sentences as a reference word. For example, as shown in Figure 24, suppose the string data of the question sentence when the word counting unit 41 obtains the selected word set of question sentences is "When transferring an incoming external call to an internal extension with product A, it becomes a one-sided call." Suppose that the word set of question sentences obtained from this string data of question sentences is "product A / external / incoming / internal / transfer / when / one-sided call".

[0147] The word occurrence counting unit 41 uses each word in the selected question word set as a base word. For example, the word occurrence counting unit 41 initially uses the word "product A" from the question word set as a base word. The word occurrence counting unit 41 uses words that exist within a predetermined number of words before and after the base word "product A" as co-occurring words of the base word and counts the number of times each co-occurring word appears within that predetermined number of words before and after the base word. Here, for example, the predetermined number of words may be set to "2". In this case, the word occurrence counting unit 41 uses the words "outbound line" and "incoming call" that exist within two words before and after "product A" as co-occurring words of the base word "product A" and counts the number of times each of the words "outbound line" and "incoming call" appears as "1". The word occurrence count unit 41 associates "external line" and "incoming call" as co-occurring words with the base word "product A," and further associates the co-occurring word "external line" with an occurrence count of "1," and the co-occurring word "incoming call" with an occurrence count of "1," generating data. The word occurrence count unit 41 outputs the generated data and the word group of the selected question sentence "product A / external line / incoming call / internal line / transfer / internal call / one-way call" to the co-occurring word data generation unit 42 (Sc4).

[0148] The co-occurring word data generation unit 42 takes in the data output by the occurrence word count unit 41 and the word group of the question sentence, and based on the taken-in data and the word group of the question sentence, generates the portion of the co-occurring word data table shown by the code 100 in Figure 24 (hereinafter referred to as the co-occurring word data table 100) shown by the code 101 in the intermediate data storage unit 20. In generating the co-occurring word data table 100, the co-occurring word data generation unit 42 records a blank "-" in the column item corresponding to the word "product A", which is the reference word shown in the taken-in data. The co-occurring word data generation unit 42 records the corresponding occurrence count "1" in the column items corresponding to the co-occurring words "external line" and "incoming call", respectively, and records a "0" in the column items corresponding to the words "internal line", "transfer", "on the other", and "one-way call", which are included in the word group of the question sentence other than the co-occurring words "external line" and "incoming call" (Sc5).

[0149] The word occurrence count unit 41 and the co-occurring word data generation unit 42 similarly perform the Sc4 and Sc5 processes for words other than "product A" in the selected question sentence word group (loop Lc2s~Lc2e). In this iterative process, records from the 2nd to the 7th row of the co-occurring word data table 100 are generated. In the co-occurring word data table 100, each row's record becomes co-occurring word data corresponding to each of the reference words.

[0150] The word occurrence count unit 41 and the co-occurrence word data generation unit 42 perform the loop Lc2s to Lc2e processing for each word group of other question sentences (loop Lc1s to Lc1e). When the word occurrence count unit 41 performs the Sc5 processing for the word group of question sentences selected from the second time onward, several co-occurrence word data are already stored in the co-occurrence word data table 100 of the intermediate data storage unit 20. Therefore, in the Sc5 processing, the co-occurrence word data generation unit 42 obtains combinations of the reference word output by the word occurrence count unit 41 through the second and subsequent Sc4 processing, the co-occurrence word, and the number of occurrences, and then determines whether or not co-occurrence word data corresponding to the reference word of the obtained combination exists in the co-occurrence word data table 100.

[0151] The co-occurring word data generation unit 42 determines that co-occurring word data corresponding to the base word of the combination does not exist in the co-occurring word data table 100. In this case, the co-occurring word data generation unit 42 adds a record for the base word, i.e., co-occurring word data, to the co-occurring word data table 100, and records the number of occurrences of the combination in the column of the co-occurring word for that combination in the added co-occurring word data.

[0152] The co-occurring word data generation unit 42 determines that co-occurring word data corresponding to the base word of the combination exists in the co-occurring word data table 100. In this case, the co-occurring word data generation unit 42 selects the co-occurring word data in the co-occurring word data table 100. If the selected co-occurring word data contains a column corresponding to the co-occurring words of the combination, the co-occurring word data generation unit 42 calculates the sum of the occurrence counts shown in that column and the occurrence count of the combination, and records the calculated sum as the occurrence count of that column. If the selected co-occurring word data does not contain a column corresponding to the co-occurring words of the combination, the co-occurring word data generation unit 42 adds a column corresponding to the co-occurring words and records the occurrence count of the combination in the added column.

[0153] For example, suppose the second processing step in the loop Lc1s~Lc1e is performed on the group of words in the question sentence "Product A can connect to internal lines, but not to external lines" obtained from the question sentence string data "Product A can connect to internal lines, but not to external lines" and the group of words in the question sentence "Product A can connect to internal lines, but not to external lines" obtained from the question sentence string data "Product A can connect to internal lines, but not to external lines" and the group of words "Product A can connect to internal lines" and "not to external respectively) is performed on the second processing step in the loop Lc1s~Lc1e is performed on the group of words in the question sentence "Product A can connect to internal lines, but not to external lines" and "not to external lines" and "not to external lines" and "not to external lines" respectively).

[0154] When the co-occurrence word data generation unit 42 finishes processing Sc5 for the last criterion word in the word group of the last question sentence, the processing of the loops Lc1s to Lc1e also finishes, and the process shown in Figure 23 ends.

[0155] (Display processing of co-occurring words by document data processing device 1b) The following describes the process by which the document data processing device 1b displays co-occurring words using the co-occurring word data stored in the intermediate data storage unit 20. Figure 26 is a flowchart showing the processing flow by the co-occurring word recommendation unit 54.

[0156] The search query acquisition unit 52 displays, for example, the search query input screen 110 shown in Figure 27, on the output device 3, where the search query input box indicated by reference numeral 111 (hereinafter referred to as the search query input box 111) is blank and nothing is displayed in the area indicated by reference numeral 112. When a user performs an operation on the input device 4 to write a string into the search query input box 111 of the search query input screen 110, the input device 4 outputs the string data of the written string to the search query acquisition unit 52. When the search query acquisition unit 52 receives the string data output by the input device 4, it displays the received string data in the search query input box 111. The search query acquisition unit 52 outputs the string data displayed in the search query input box 111 as a search query to the co-occurring word recommendation unit 54.

[0157] The co-occurring word recommendation unit 54 takes in the search query output by the search query acquisition unit 52 (Sd1). The co-occurring word recommendation unit 54 outputs a word extraction request signal containing the taken-in search query to the word extraction unit 12a. When the word extraction unit 12a receives the word extraction request signal from the co-occurring word recommendation unit 54, it extracts words from the search query contained in the received word extraction request signal and outputs a word extraction completion signal containing a list of the extracted words to the co-occurring word recommendation unit 54. The co-occurring word recommendation unit 54 takes in the word extraction completion signal output by the word extraction unit 12a (Sd2). The co-occurring word recommendation unit 54 detects all of the co-occurring word data from the co-occurring word data table 100 in the intermediate data storage unit 20, using each of the words shown in the word group contained in the taken-in word extraction completion signal as a reference word (Sd3).

[0158] For example, suppose the string written in the search query input field 111 is "Transfer with product A", and the contents of the co-occurring word data table 100 stored in the intermediate data storage unit 20 are as shown in Figure 25. In this case, the co-occurring word recommendation unit 54 takes in a word extraction completion signal in processing Sd2 that includes a group of words listing the words "product A" and "transfer" extracted by the word extraction unit 12a from the string "Transfer with product A". In processing Sd3, the co-occurring word recommendation unit 54 detects the records in the co-occurring word data table 100, which use the words "product A" and "transfer" from the group of words included in the taken-in word extraction completion signal as reference words, as co-occurring word data for the reference words "product A" and "transfer". Here, the co-occurrence word recommendation unit 54 detects the record in the first row of the co-occurrence word data table 100 as the co-occurrence word data for the base word "product A", and detects the record in the fifth row of the co-occurrence word data table 100 as the co-occurrence word data for the base word "transfer".

[0159] The co-occurrence word recommendation unit 54 calculates the total number of occurrences of common co-occurrence words in all detected co-occurrence word data and uses that total as the occurrence count of the common co-occurrence word (Sd4). The co-occurrence word recommendation unit 54 then sorts each of the co-occurrence words included in all detected co-occurrence word data in descending order of their respective occurrence counts (Sd5).

[0160] The co-occurrence word data detected by the co-occurrence word recommendation unit 54 for each of the base words "product A" and "transfer" is described in the format "co-occurrence word / frequency of occurrence" as follows: For the base word "product A", the co-occurrence words are "external line / 2", "incoming call / 1", "internal line / 2", and "call / 1". For the base word "transfer", the co-occurrence words are "incoming call / 1", "internal line / 1", "internal line / 1", and "one-way call / 1". Note that words with a frequency of occurrence of "0" are not co-occurrence words, so the co-occurrence word recommendation unit 54 does not add such words to the list. Also, if any of the base words are included in the co-occurrence words, the co-occurrence word recommendation unit 54 excludes those co-occurrence words. The co-occurring word recommendation unit 54 calculates the total number of occurrences of common co-occurring words, and when these are arranged in descending order of frequency, the order is as shown in Figure 28: "Extension / 3", "External / 2", "Incoming Call / 2", "Internal / 1", "One-way Call / 1", and "Call / 1".

[0161] The co-occurring word recommendation unit 54 selects co-occurring words up to a predetermined position in the sequence of co-occurring words, outputs the selected co-occurring words to the search query acquisition unit 52 (Sd6), and terminates processing. Here, let's assume that "3" is predetermined as the predetermined position. In this case, the co-occurring word recommendation unit 54 arranges the top 3 co-occurring words in descending order of frequency of occurrence to generate a word list of "internal line," "external line," and "incoming call," and outputs the generated word list to the search query acquisition unit 52. When the search query acquisition unit 52 receives the word list output by the co-occurring word recommendation unit 54, it displays the three words arranged in the order of "internal line," "external line," and "incoming call" shown in the received word list on the search query input screen 110 displayed on the output device 3, as indicated by the reference numeral 112 in Figure 27.

[0162] As indicated by symbol 112, the three co-occurring words "extension," "external line," and "incoming call" are displayed in a state where the user can select one, and the user performs the operation of selecting "extension" to the input device 4. In this case, the input device 4 outputs a search query addition signal containing the selected word "extension" to the search query acquisition unit 52. The search query acquisition unit 52 takes in the search query addition signal containing the word "extension" output by the input device 4. When the search query acquisition unit 52 takes in the search query addition signal, as shown in Figure 29, it adds a space after the already displayed "transfer on product A" in the search query input frame 111, and then adds the word "extension" contained in the acquired search query addition signal and displays it. The search query acquisition unit 52 outputs the string data displayed in the search query input frame 111, "transfer on product A extension," as a search query to the co-occurring word recommendation unit 54.

[0163] The co-occurring word recommendation unit 54 performs the Sd2 process on the search query output by the search query acquisition unit 52, thereby obtaining the three words "product A," "transfer," and "extension" from the word extraction completion signal output by the word extraction unit 12a. In the Sd3 process, the co-occurring word recommendation unit 54 detects the records in the first, fifth, and fourth rows of the co-occurring word data table 100, which uses "product A," "transfer," and "extension" as base words, respectively, as co-occurring word data for the base words "product A," "transfer," and "extension."

[0164] Here, the co-occurrence word data for the base word "internal extension" consists of "product A / 2", "external line / 3", "incoming call / 1", "transfer / 1", "internal / 1", and "call / 1". In processing Sd4, the co-occurrence word recommendation unit 54 calculates the total number of occurrences of co-occurrence words common to the three co-occurrence word data. In this case, "external line", "incoming call", "internal line", "internal", and "call" are common. However, since "internal line" is the base word, the co-occurrence word recommendation unit 54 excludes "internal line" and then calculates a total value of "5" for "external line", a total value of "3" for "incoming call", a total value of "2" for "internal", and a total value of "2" for "call".

[0165] In processing Sd5, the co-occurring word recommendation unit 54 sorts all co-occurring words, "external line," "incoming call," "on the other," "one-way call," and "call," from the three detected co-occurring word data, excluding the occurrence counts corresponding to the base words "product A," "transfer," and "extension," in descending order of occurrence. In this case, as shown in Figure 30, the co-occurring word recommendation unit 54 sorts them in the order of "external line / 5," "incoming call / 3," "on the other / 2," "call / 2," and "one-way call / 1." In processing Sd6, the co-occurring word recommendation unit 54 sorts the top three co-occurring words in descending order of occurrence to generate a word list of "external line," "incoming call," "on the other," and "call," and outputs the generated word list to the search query acquisition unit 52. The search query acquisition unit 52 takes in the word list output by the co-occurring word recommendation unit 54. The search query acquisition unit 52, on the search query input screen 110 displayed on the output device 3, displays four co-occurring words arranged in the order of "external line," "incoming call," "on the occasion," and "call" as shown by reference numeral 112a in Figure 29, instead of displaying the co-occurring words shown by reference numeral 112 in Figure 27.

[0166] Assuming that the search query input screen 110 shown in Figure 29 is displayed on the output device 3, for example, a user selects a radio button corresponding to an AND search in the section indicated by reference numeral 114, and then selects the search button 113 on the input device 4. The input device 4 outputs a search request signal to the search processing unit 53 that includes a search condition indicating an AND search and a search query, which is the string data "Transfer extension for product A" displayed in the search query input box 111. When the search processing unit 53 receives the search request signal from the input device 4, it splits the string data indicated by the search query included in the search request signal by a space, resulting in two string data: "Transfer for product A" and "Extension". According to the AND search condition included in the search request signal, the search processing unit 53 detects response data from the response data storage unit 2 that includes the string data of "Transfer for product A" and "Extension" in one of the parts of the string data of the question sentence shown in the "Question" item of the response data. The search processing unit 53 outputs the detected response data to the output device 3.

[0167] (Effects of the document data processing device 1b) Searching the response data storage unit 2 using more appropriate words allows for the detection of response data with a higher accuracy rate, i.e., more appropriate answers. In contrast, in primary reception contact centers, it is necessary to answer the questioner quickly, and the time required to research the content of the answer is limited. Therefore, when a respondent searches past response data, the number of words they can think of in their search terms is limited to only a few words. In such cases, by using the document data processing device 1b, co-occurring word data can be generated in advance from the response data, and based on the generated co-occurring word data, words that frequently appear together with the words included in the search terms thought of by the respondent in the response data can be presented to the respondent as co-occurring words. Therefore, by adding the presented co-occurring words to the search, even if the search terms thought of by the respondent do not contain enough information to obtain the desired response data, the amount of information can be supplemented, thereby improving the probability of obtaining the desired response data without reducing search accuracy, and enabling more appropriate answers to be given to the questioner.

[0168] (Other configuration examples of document data processing device 1b) In the document data processing device 1b, the word occurrence count unit 41 has predetermined items designated as "questions," but these predetermined items may also be "answers," or they may be items that are both "questions" and "answers."

[0169] In the example shown, the predetermined number of words in the word counting unit 41 of the document data processing device 1b is set to "2". However, the predetermined number of words can be any integer value of 1 or greater. Alternatively, the word counting unit 41 may set the predetermined number of words to, for example, the maximum number of characters that can be written in an item, and count the occurrence count of all words that exist before and after the reference word as co-occurring words.

[0170] In the co-occurrence word recommendation unit 54 of the document data processing device 1b, an example was shown where the predetermined upper position is set to "3," but the predetermined position can be any integer value greater than or equal to 1. Furthermore, the co-occurrence word recommendation unit 54 may use any method for selecting co-occurrence words based on their frequency of occurrence, such as selecting co-occurrence words whose frequency of occurrence is greater than or equal to a predetermined certain value to generate a word list.

[0171] In the document data processing device 1b, the co-occurring word data generation unit 42 may, upon completion of processing in the loops Lc1s to Lc1e shown in Figure 23, count the number of response data to be processed, and rewrite the occurrence count of the co-occurring word data by dividing the occurrence count of each co-occurring word shown in each co-occurring word data by the counted number of response data. This makes it possible to standardize the scale of occurrence counts for each co-occurring word, i.e., to reduce the impact when a particular word appears many times in a specific sentence of response data, thus enabling a simplified normalization.

[0172] In the document data processing device 1b, the search processing unit 53 performs a search targeting the response data stored in the response data storage unit 2. However, it may also perform a search targeting any system or device that stores information related to the response data, such as a search targeting an AI or machine learning system that has been trained using the response data stored in the response data storage unit 2 as training data, or a search using the device disclosed in Japanese Patent Application Publication No. 2024-132249.

[0173] The document data processing device 1b may be configured such that the intermediate data storage unit 20 does not pre-store word data generated by the document data processing devices 1 and 1a, and the occurrence word count unit 41 starts processing from Sc3 without performing the processing of Sc1 and Sc2 in Figure 23. In other words, the document data processing device 1b may exist as a completely independent device from the document data processing devices 1 and 1a.

[0174] (Other configuration examples common to document data processing devices 1, 1a, and 1b) The document data processing devices 1, 1a, and 1b perform processing on the response data shown in Figure 2. However, they may also perform the above-described processing on any document data other than the response data, which has at least a "Case ID" field and one or more fields indicating string data, by replacing the "Question" or "Answer" fields with string data indicated in any field of the document data.

[0175] <Hardware Configuration> Figure 31 shows an example of the hardware configuration of the document data processing devices 1, 1a, and 1b according to this disclosure. The document data processing devices 1, 1a, and 1b according to this disclosure are computers comprising, for example, a CPU (Central Processing Unit) 201, RAM (Random Access Memory) 202, ROM (Read Only Memory) 203, auxiliary storage device 204, and interface module 205. The CPU 201, RAM 202, ROM 203, auxiliary storage device 204, and interface module 205 are interconnected by a bus 206. The auxiliary storage device 204 is, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The interface module 205 is connected to a response data storage unit 2, an output device 3, and an input device 4.

[0176] An application program pre-stored in ROM 203 or auxiliary storage device 204 is executed by CPU 201, thereby configuring acquisition units 11, 11a, word extraction units 12, 12a, vectorization unit 13, similarity calculation units 14, 14a, high-similarity data selection unit 16, transition relationship-considering group determination unit 17, community detection unit 18, group combination adjustment unit 19, group representative data generation unit 31, group feature analysis unit 32, occurrence word count unit 41, co-occurring word data generation unit 42, search query acquisition unit 52, search processing unit 53, and co-occurring word recommendation unit 54, and a storage area for the intermediate data storage unit 20 is secured in RAM 202 or auxiliary storage device 204.

[0177] <Fourth Embodiment> Hereinafter, an embodiment relating to the present disclosure will be described with reference to the figures. As shown in Figure 32, the document data processing device 300 includes an acquisition means 301 for acquiring document data having one or more items, a word extraction means 303 for extracting words from string data indicated in the items of the document data, a vectorization means 303 for vectorizing each of the string data based on the words in each string data, a similarity calculation means 304 for calculating the similarity between string data of the same item based on the vectors corresponding to each of the string data, and a grouping means 305 for selecting document data that have a direct and indirect linking relationship based on the similarity and a predetermined similarity threshold, and grouping each of the selected combinations with a linking relationship.

[0178] As shown in Figure 33, the acquisition means 301 acquires document data having one or more items (S301). The word extraction means 302 extracts words from the string data indicated in the items of the document data acquired by the acquisition means 301 (S302). The vectorization means 303 vectorizes each of the string data based on the words extracted by the word extraction means 302 (S303). The similarity calculation means 304 calculates the similarity between string data with the same items based on the vectors corresponding to each of the string data (S304). The grouping means 305 selects document data that have direct and indirect coherence relationships based on the similarity calculated by the similarity calculation means 304 and a predetermined similarity threshold, and groups each of the selected combinations with coherence relationships (S305).

[0179] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure are possible, as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0180] Some or all of the embodiments described above may also be described as follows, but are not limited to the following.

[0181] (Note 1) A document data processing device comprising: acquisition means (e.g., acquisition units 11, 11a) for acquiring document data having one or more items (e.g., correspondence data); word extraction means (e.g., word extraction units 12, 12a) for extracting words from string data indicated in the items of the document data; vectorization means (e.g., vectorization unit 13) for vectorizing each of the string data based on the words in each of the string data; similarity calculation means (e.g., similarity calculation units 14, 14a) for calculating the similarity between string data having the same items based on the vectors corresponding to each of the string data; and grouping means (e.g., grouping unit 15) for selecting document data that have a direct and indirect association relationship based on the similarity and a predetermined similarity threshold, and grouping each of the selected combinations with the association relationship into a group (e.g., a transition relationship-considering group, or a belonging group).

[0182] (Note 2) The document data processing apparatus according to (Note 1), further comprising a group representative data generation means (for example, a group representative data generation unit 31) that generates representative document data for each group from the document data belonging to each of the groups.

[0183] (Note 3) A document data processing device according to (Note 1) or (Note 2), comprising: a group feature analysis means (for example, a group feature analysis unit 32) that calculates the centroid vector of each of the groups and, based on the calculated centroid vectors, calculates the distance between the groups and the contribution of words common to the groups.

[0184] (Note 4) The system includes a group combination adjustment means (for example, a group combination adjustment unit 19), the acquisition means acquires incremental document data (for example, incremental correspondence data), which is the document data to be added; the word extraction means extracts words from the string data shown in the items of the incremental document data; the vectorization means vectorizes each of the string data based on the word extracted by the word extraction means from the incremental document data; the similarity calculation means (for example, a similarity calculation unit 14a) calculates the similarity between string data with the same items based on the vectors corresponding to each of the string data of the incremental document data; and calculates the similarity between the string data of the incremental document data and the string data of the existing document data with the same items based on the vectors corresponding to each of the string data of the incremental document data and the vectors corresponding to each of the string data of the existing document data, which is the document data before the incremental document data was added. A document data processing device according to any one of (Note 1) to (Note 3), wherein the grouping means selects the incremental document data and the existing document data that have a direct and indirect linking relationship based on the similarity and the similarity threshold, and designates each of the selected combinations with a linking relationship as an incremental group (e.g., an incremental transition relationship-considering group), and the group linking adjustment means detects, for each of the incremental document data belonging to the incremental group, the number of links between the incremental document data and the group formed by the existing document data (e.g., a transition relationship-considering group) and the number of links within the incremental group, indicated by the number of other incremental document data to which the incremental document data is linked, and selects the destination to which each of the incremental document data belongs based on the detected number of links and the group formed by the existing document data, and regroups the group formed by the existing document data and the incremental group according to the selection.

[0185] (Note 5) The document data processing device according to any one of (Note 1) to (Note 4), wherein the grouping means regroups the groups by dividing them by community detection for each of the groups.

[0186] (Note 6) A document data processing device according to any one of (Note 1) to (Note 5), comprising: an occurrence word counting means (e.g., occurrence word counting unit 41) that selects each of the words shown in a word group including a word extracted by the word extraction means from the string data corresponding to a predetermined item in advance as a reference word, and for each word group from which the reference word was selected, the words that exist before and after the reference word are considered co-occurring words of the reference word, and counts the number of times the co-occurring words appear in the word group; and a co-occurring word data generation means (e.g., co-occurring word data generation unit 42) that generates co-occurring word data by associating the total number of occurrences counted by the occurrence word counting means for each word group for the combination of the reference word and the co-occurring word corresponding to the reference word with the combination of the reference word and the co-occurring word.

[0187] (Note 7) A document data processing device according to (Note 6), comprising: a search query acquisition means (for example, a search query acquisition unit 52) ​​that acquires a search query provided from an external source; and a co-occurring word recommendation means (for example, a co-occurring word recommendation unit 54) that detects co-occurring word data with each of the words indicated by the search query as the reference word, and if the detected co-occurring word data contains a common co-occurring word, the sum of the occurrence counts of the common co-occurring word is taken as the occurrence count of the common co-occurring word, and outputs each of the co-occurring words included in the detected co-occurring word data according to the occurrence count.

[0188] (Note 8) The document data processing device according to (Note 7), wherein the co-occurring word recommendation means detects co-occurring word data using the word indicated by the search query and the selected co-occurring word as the reference words when any of the co-occurring words output by itself is selected.

[0189] (Note 9) A document data processing method comprising: acquiring document data having one or more items; extracting words from string data indicated in the items of the acquired document data; vectorizing each of the string data based on the extracted words; calculating the similarity between string data having the same items based on the vectors corresponding to each of the string data; selecting document data that have direct and indirect coherence relationships based on the calculated similarity and a predetermined similarity threshold; and grouping each of the selected combinations with coherence relationships.

[0190] (Appendix 10) The document data processing method described in (Appendix 9), which generates representative document data for each group from the document data belonging to each of the groups.

[0191] (Note 11) The document data processing method according to (Note 9) or (Note 10), which involves calculating the centroid vector of each of the groups, and based on the calculated centroid vectors, calculating the distance between the groups and the contribution of words common to the groups.

[0192] (Note 12) The incremental document data, which is the document data to be added, is acquired; words are extracted from the string data shown in the items of the acquired incremental document data; each of the string data is vectorized based on the words of each of the extracted string data; the similarity between string data with the same item is calculated based on the vectors corresponding to each of the string data of the incremental document data; the similarity between the string data of the incremental document data and the string data of the existing document data with the same item is calculated based on the vectors corresponding to each of the string data of the incremental document data and the vectors corresponding to each of the string data of the existing document data, which is the document data before the incremental document data was added; and based on the calculated similarity and the similarity threshold, directly A document data processing method according to any one of (Appendix 9) to (Appendix 11): Select the incremental document data and the existing document data that have an immediate and indirect linking relationship; designate each of the selected combinations with the linking relationship as an incremental group; for each of the incremental document data belonging to the incremental group, detect the number of links between the incremental document data and the group formed by the existing document data and the number of links within the incremental group, which is indicated by the number of other incremental document data to which the incremental document data is linked; select the destination to which each of the incremental document data belongs based on the detected number of links and the group formed by the existing document data; and regroup the group formed by the existing document data and the incremental group according to the selection.

[0193] (Note 13) A document data processing method according to any one of (Note 9) to (Note 12), wherein the group is regrouped by dividing the group by community detection for each of the groups.

[0194] (Note 14) A document data processing method according to any one of (Note 9) to (Note 13), comprising: selecting each of the words shown in a group of words including a word extracted from the string data corresponding to a predetermined item as a reference word; for each group of words from which the selected reference word was derived, the words before and after the reference word are treated as co-occurring words of the reference word; counting the number of occurrences of the co-occurring words in the group of words; and generating co-occurring word data by associating the total number of occurrences counted for each group of words with the combination of the reference word and the co-occurring word corresponding to the reference word.

[0195] (Note 15) The document data processing method described in (Note 14), which involves obtaining a search query provided from an external source, detecting co-occurring word data with each of the words indicated by the obtained search query as the reference word, and if the detected co-occurring word data contains a common co-occurring word, taking the total number of occurrences of the common co-occurring word as the occurrence count of the common co-occurring word, and outputting each of the co-occurring words included in the detected co-occurring word data according to the occurrence count.

[0196] (Appendix 16) The document data processing method described in (Appendix 15), wherein if any of the outputted co-occurring words is selected, the co-occurring word data is detected with the word indicated by the search query and the selected co-occurring word each as the reference word.

[0197] (Note 17) A program for causing a computer to function as an acquisition means for acquiring document data having one or more items, a word extraction means for extracting words from string data indicated in the items of the document data, a vectorization means for vectorizing each of the string data based on the words in each of the string data, a similarity calculation means for calculating the similarity between string data having the same items based on the vectors corresponding to each of the string data, and a grouping means for selecting document data that have a direct and indirect association relationship based on the similarity and a predetermined similarity threshold, and grouping each of the selected combinations with the association relationship.

[0198] (Appendix 18) The program described in (Appendix 17) for functioning as a group representative data generation means that generates representative document data for each group from the document data belonging to each of the groups.

[0199] (Note 19) The program described in (Note 17) or (Note 18) for functioning as a group feature analysis means that calculates the centroid vector of each of the groups and, based on the calculated centroid vectors, calculates the distance between the groups and the contribution of words common to the groups.

[0200] (Note 20) Functioning as a group joining adjustment means, the acquisition means acquires incremental document data which is the document data to be added, the word extraction means extracts words from the string data shown in the items of the incremental document data, the vectorization means vectorizes each of the string data based on the word for each string data extracted by the word extraction means from the incremental document data, the similarity calculation means calculates the similarity between string data with the same item based on the vector corresponding to each of the string data of the incremental document data, and calculates the similarity between the string data of the incremental document data and the string data of the existing document data with the same item based on the vector corresponding to each of the string data of the incremental document data and the vector corresponding to each of the string data of the existing document data which is the document data before the incremental document data was added. The program as described in any one of (Note 17) to (Note 19), wherein the grouping means selects the incremental document data and the existing document data that have a direct and indirect linking relationship based on the similarity and the similarity threshold, and designates each of the selected combinations with a linking relationship as an incremental group, and the group linking adjustment means detects, for each of the incremental document data belonging to the incremental group, the number of links between the incremental document data and the group formed by the existing document data and the number of links within the incremental group indicated by the number of other incremental document data to which the incremental document data is linked, and selects the destination to which each of the incremental document data belongs based on the detected number of links and the group formed by the existing document data, and regroups the group formed by the existing document data and the incremental group according to the selection.

[0201] (Note 21) The grouping means is a program according to any one of (Note 17) to (Note 20), which regroups the groups by dividing them by community detection for each of the groups.

[0202] (Note 22) A program described in any one of (Note 17) to (Note 21) for functioning as: a word counting means that selects each of the words shown in a word group containing words extracted from the string data corresponding to a predetermined item by the word extraction means as a reference word, and for each word group from which the reference word was selected, the words that exist before and after the reference word are considered co-occurring words of the reference word, and counts the number of times the co-occurring words appear in the word group; and a co-occurring word data generation means that generates co-occurring word data by associating the total number of occurrences counted by the word counting means for each word group for the combination of the reference word and the co-occurring word corresponding to the reference word with the combination of the reference word and the co-occurring word.

[0203] (Note 23) The program described in (Note 22) for functioning as a search query acquisition means for acquiring a search query provided from an external source, a co-occurring word recommendation means for detecting co-occurring word data using each of the words indicated by the search query as a reference word, and if the detected co-occurring word data contains a common co-occurring word, taking the total number of occurrences of the common co-occurring word as the occurrence count of the common co-occurring word, and outputting each of the co-occurring words included in the detected co-occurring word data according to the occurrence count.

[0204] (Note 24) The program described in (Note 23) wherein the co-occurring word recommendation means detects the co-occurring word data, with the word indicated by the search query and the selected co-occurring word each being the reference word, when any of the co-occurring words output by itself is selected.

[0205] (Note 25) A document data processing device comprising: an acquisition means (e.g., an acquisition unit 11a) for acquiring document data; a word extraction means (e.g., an acquisition unit 11a) for extracting words from string data contained in the document data and generating a group of words for each string data; an occurrence word counting means (e.g., an occurrence word counting unit 41) for selecting each of the words contained in the group of words as a reference word, and for each group of words from which the reference word was selected, for words that exist before and after the reference word as co-occurring words of the reference word, and for counting the number of times the co-occurring words appear in the group of words; and a co-occurring word data generation means (e.g., a co-occurring word data generation unit 42) for generating co-occurring word data by associating the total number of occurrences counted by the occurrence word counting means for each group of words for the combination of the reference word and the co-occurring word with the combination of the reference word and the co-occurring word.

[0206] (Note 26) A document data processing method comprising: acquiring document data; extracting words from string data contained in the acquired document data to generate word groups for each string data; selecting each word in the generated word group as a reference word; for each word group from which the selected reference word was derived, the words that exist before and after the reference word are considered co-occurring words of the reference word; counting the number of occurrences of the co-occurring words in the word group; and generating co-occurring word data by associating the total number of occurrences counted for each word group with the combination of the reference word and the corresponding co-occurring word.

[0207] (Note 27) A program for causing a computer to function as an acquisition means for acquiring document data, a word extraction means for extracting words from string data contained in the document data and generating a group of words for each string data, an occurrence word counting means for selecting each word contained in the group of words generated by the word extraction means as a reference word, and for each group of words from which the reference word was selected, the words that exist before and after the reference word are considered co-occurring words of the reference word, and counting the number of times the co-occurring words appear in the group of words, and a co-occurring word data generation means for generating co-occurring word data by associating the total number of occurrences counted by the occurrence word counting means for each group of words for the combination of the reference word and the co-occurring word corresponding to the reference word with the combination of the reference word and the co-occurring word. [Explanation of symbols]

[0208] 1. Document data processing device 2. Response data storage unit 3. Output device 11,11a Acquisition part 12,12a Word extraction section 13 Vectorization section 14,14a Similarity calculation part 15 Grouping Department 16. High Similarity Data Selection Section 17. Group Determination Section Considering Transitional Relationships 18 Community Discovery Department 19 Group coupling adjustment unit 20 Intermediate data storage unit 31 Group Representative Data Generation Department 32 Group Characteristics Analysis Department 41. Word Count Section 42. Co-occurrence word data generation unit 51 Search Section 52 Search Query Retrieval Unit 53 Search Processing Unit 54. Recommended Co-occurring Words

Claims

1. A means for acquiring document data having one or more items, A word extraction means for extracting words from the string data shown in the item of the document data, A vectorization means that vectorizes each of the string data based on the words in each of the string data, A similarity calculation means that calculates the similarity between string data that have the same item based on the vector corresponding to each of the string data, A grouping means that selects document data that have a direct and indirect linking relationship based on the aforementioned similarity and a predetermined similarity threshold, and groups each of the selected combinations of linking relationships, A document data processing device equipped with the following features.

2. Group representative data generation means that generates representative document data for each group from the document data belonging to each of the groups. A document data processing device according to claim 1, comprising:

3. A group feature analysis means that calculates the centroid vector of each of the aforementioned groups, and calculates the distance between the groups and the contribution of common words between the groups based on the calculated centroid vectors. A document data processing device according to claim 1, comprising:

4. Equipped with group coupling adjustment means, The acquisition means is, The incremental document data, which is the document data to be added, is obtained. The word extraction means is Words are extracted from the string data shown in the item of the incremental document data, The vectorization means is Based on the word for each string data extracted by the word extraction means from the incremental document data, each of the string data is vectorized. The similarity calculation means is, Based on the vectors corresponding to each of the string data in the incremental document data, the similarity between string data with the same item is calculated; and based on the vectors corresponding to each of the string data in the incremental document data and the vectors corresponding to each of the string data in the existing document data which is the document data before the incremental document data was added, the similarity between the string data in the incremental document data and the string data in the existing document data with the same item is calculated. The grouping means is, Based on the similarity and the similarity threshold, the incremental document data and the existing document data that have a direct and indirect linking relationship are selected, and each of the selected combinations with the linking relationship is designated as an incremental group. The group coupling adjustment means is For each of the incremental document data belonging to the incremental group, the number of connections between the incremental document data and the group formed by the existing document data, and the number of connections within the incremental group, indicated by the number of other incremental document data to which the incremental document data is connected, are detected. Based on the detected number of connections and the group formed by the existing document data, a destination is selected to which each of the incremental document data belongs, and the group formed by the existing document data and the incremental group are regrouped according to the selection. The document data processing device according to claim 1.

5. The grouping means is, The groups are regrouped by dividing them by community detection for each of the groups. A document data processing device according to any one of claims 1 to 4.

6. The word extraction means selects each of the words shown in the word group containing the words extracted from the string data corresponding to a predetermined item as a reference word, and for each word group from which the reference word was selected, the words that exist before and after the reference word are considered co-occurring words of the reference word, and the word occurrence count means counts the number of times the co-occurring words appear in the word group, Co-occurring word data generation means generates co-occurring word data by associating the total number of occurrences counted by the occurrence word counting means for each word group with the combination of the reference word and the co-occurring word corresponding to the reference word, A document data processing device according to claim 1, comprising:

7. A search query acquisition method that obtains search queries provided from an external source, A co-occurring word recommendation means that detects co-occurring word data using each of the words indicated by the search query as the reference word, and if the detected co-occurring word data contains a common co-occurring word, the sum of the occurrence counts of the common co-occurring word is taken as the occurrence count of the common co-occurring word, and outputs each of the co-occurring words included in the detected co-occurring word data according to the occurrence count, A document data processing device according to claim 6, comprising:

8. The aforementioned co-occurring word recommendation means is If any of the co-occurring words output by itself is selected, the co-occurring word data is detected with the word indicated by the search query and the selected co-occurring word as the reference words. The document data processing device according to claim 7.

9. Retrieve document data that has one or more items, From the string data shown in the item of the acquired document data, extract a word. Based on the words in each of the extracted string data, each of the string data is vectorized. Based on the vector corresponding to each of the aforementioned string data, the similarity between the aforementioned string data whose items are the same is calculated. Based on the calculated similarity and a predetermined similarity threshold, the document data that have a direct and indirect relationship are selected, and each of the selected combinations with such a relationship is grouped together. Document data processing method.

10. Computers, Acquisition means for acquiring document data having one or more items, A word extraction means for extracting words from the string data shown in the item of the document data, A vectorization means that vectorizes each of the string data based on the words in each of the string data, Similarity calculation means calculates the similarity between string data that have the same item, based on the vector corresponding to each of the string data. A grouping means that selects document data that have a direct and indirect linking relationship based on the aforementioned similarity and a predetermined similarity threshold, and groups each of the selected combinations of linked documents. A program designed to function as such.

Images