Processing method, processing program, and information processing device

Abstract

Upon receiving a plurality of characters or character strings to be registered in a database, this information processing device calculates a plurality of vectors corresponding respectively to the plurality of characters or character strings. The information processing device classifies the calculated plurality of vectors into a plurality of clusters according to the distance between the vectors. The information processing device obtains, for each of the plurality of clusters, a representative vector representing the one or plurality of vectors included in the cluster, and the distance from the representative vector of each cluster that corresponds to the size of the one or plurality of vectors included in the cluster. The information processing device stores, in a storage unit, dictionary information in which the calculated representative vectors and the obtained distances are associated with cluster identification information identifying each cluster.

Description

[Technical field]

[0001] The present invention relates to a processing method and the like. [Background technology]

[0002] A huge amount of data such as text is stored in a database (DB), and it is required to appropriately search for data similar to an input query specified by a user from the DB.

[0003] In the conventional technology, when an input query is received, a data search is performed using a previously set inverted index. In the prior art, when each text is registered in a DB, a morphological analysis is performed and an inverted index is generated by associating each word with the position information of the corresponding text in the DB. This process is repeatedly executed every time a new text is registered in the DB. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-111479 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the above-mentioned conventional technology has a problem in that the load of data registration processing is large for AI (Artificial Intelligence) analysis that supports multi-dimensional vectors.

[0006] In AI analysis, when registering data such as text consisting of character strings, chemical structural formulas of organic compounds, and genome sequences, it is necessary to assign and accumulate vectors to meaningful units such as words, functional groups, sentences, and primary structures of functional groups. However, taking text as an example, there are generally about a million different types of words, and it is difficult to register the vectors and their positions of each word and sentence in all the text registered in the DB in association with the inverted index. For this reason, it is necessary to narrow down the number of vectors to be registered in the inverted index, but it takes a lot of time to identify vectors that are effective for search processing from a huge number of vectors.

[0007] In one aspect, the present invention has an object to provide a processing method, a processing program, and an information processing device that can reduce the load of data registration processing. [Means for solving the problem]

[0008] In the first proposal, a computer executes the following process. When the computer receives a plurality of characters or character strings to be registered in a database, the computer calculates a plurality of vectors corresponding to each of the plurality of characters or character strings. The computer classifies the calculated plurality of vectors into a plurality of clusters based on the inter-vector distance. For each of the plurality of clusters, the computer calculates a representative vector representing one or more vectors included in the cluster, and a distance from the representative vector of each cluster according to the magnitude of one or more vectors included in the cluster. The computer stores in a storage unit dictionary information that associates the calculated representative vector with the calculated distance for cluster identification information that identifies each cluster. Effect of the Invention

[0009] The load of data registration processing can be reduced. [Brief description of the drawings]

[0010] [Figure 1]FIG. 1 is a diagram for explaining a preparation phase process of the information processing device according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a data structure of the sentence vector dictionary. [Diagram 3] FIG. 3 is a diagram for explaining the process of the registration phase of the information processing device according to the first embodiment. [Figure 4] FIG. 4 is a diagram for explaining the process of the search phase of the information processing device according to the first embodiment. [Diagram 5] FIG. 5 is a functional block diagram of the information processing device according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a data structure of the cluster table according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of the cluster data. [Figure 8] FIG. 8 is a diagram illustrating an example of a data structure of the word vector dictionary. [Figure 9] FIG. 9 is a diagram (1) for explaining the process of calculating the sentence vector. [Figure 10] FIG. 10 is a diagram (2) for explaining the process of calculating the sentence vector. [Figure 11] FIG. 11 is a diagram (1) for explaining the process of identifying a cluster ID. [Figure 12] FIG. 12 is a diagram (2) for explaining the process of identifying a cluster ID. [Figure 13] FIG. 13 is a flowchart showing the procedure of the preparation process. [Figure 14] FIG. 14 is a flowchart showing the procedure of the search process. [Figure 15] FIG. 15 is a functional block diagram of an information processing device according to the second embodiment. As shown in FIG. [Figure 16] FIG. 16 is a diagram showing an example of the second word transposed index. [Figure 17] FIG. 17 is a diagram for explaining another application example of the sentence transposition index TS. [Figure 18] FIG. 18 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing apparatus of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of a processing method, a processing program, and an information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to these embodiments. EXAMPLES

[0012] The information processing device according to the first embodiment executes a preparation phase process, a registration phase process, and a search phase process in this order. The preparation phase process and the search phase process executed by the information processing device will be described below in this order.

[0013] Fig. 1 is a diagram for explaining the processing of the preparation phase of the information processing device according to the first embodiment. In the first embodiment, as an example, a sentence having a plurality of words will be explained as a text. In the explanation of Fig. 1, the explanation will be given using sentence 1, sentence 2, sentence 3, and sentence 4. Note that the processing of the preparation phase is performed on a population including a large number of sentences.

[0014] Let sentence 1 be "Horses like carrots." Let sentence 2 be "Carrots are horses' favorite food." Let sentence 3 be "The weather is fine today." Let sentence 4 be "The cherry blossoms have bloomed." Note that sentences 1 and 2 have swapped subjects (S) and objects (O), but they have the same meaning.

[0015] When the information processing device receives information on multiple sentences to be registered in the DB, it calculates the vector of each sentence. The information processing device calculates the vector of a sentence by accumulating the vector of each word included in the sentence. In the following explanation, the vector of a sentence is referred to as a "sentence vector." The vector of each word is defined in a word vector dictionary. The vector of a word is referred to as a "word vector."

[0016] The sentence vector of sentence 1 is "SV1". The sentence vector of sentence 2 is "SV2". The sentence vector of sentence 3 is "SV3". The sentence vector of sentence 4 is "SV4". In this Example 1, as an example, the sentence vector is described as having seven dimensions. The vector of the n-th component of the sentence vector is represented as sentence vector (n) (n=1 to 7).

[0017] The information processing device classifies each sentence into a plurality of clusters by performing clustering based on the sentence vector of each sentence. The clusters are arranged in a seven-dimensional feature space 1. A cluster ID (identification) that uniquely identifies the cluster is set for each cluster. In the example shown in FIG. 1, sentences 1 and 2 are classified into a cluster with a cluster ID of "Cr1". Sentence 3 is classified into a cluster with a cluster ID of "Cr2". Sentence 4 is classified into a cluster with a cluster ID of "Cr3".

[0018] In the following description, the clusters with cluster IDs "Cr1" to "Cr3" are denoted as clusters Cr1 to Cr3, respectively. Although clusters Cr1 to Cr3 are shown in Fig. 1, the feature space 1 may include other clusters.

[0019] The information processing device generates a sentence vector dictionary based on the clustering result. Fig. 2 is a diagram showing an example of the data structure of the sentence vector dictionary. As shown in Fig. 2, the sentence vector dictionary 50 associates a cluster ID, a representative vector, and a radius.

[0020] The cluster ID is information that uniquely identifies a cluster. The representative vector is a vector that represents the sentence vectors contained in the same cluster. There are representative vectors (1) to (7). The representative vector (n) is a vector that corresponds to the center (center of gravity) of the sentence vector (n) that belongs to the same cluster (n = 1 to 7). The diameter indicates the diameter (length of a side) of the sphere (cube) of the cluster.

[0021] 3 is a diagram for explaining the process of the registration phase of the information processing device according to the embodiment 1. When the information processing device receives an input text 2 to be registered in the DB 141, it generates a sentence transposition index TS based on each sentence included in the input text 2 and the sentence vector dictionary 50.

[0022] In the example described in Figure 3, input text 2 is assumed to be "I went to the ranch with my child. Ponies like carrots. I gave them carrots as a reward for riding the horse." For ease of explanation, of the sentences contained in input text 2, "I went to the ranch with my child" will be represented as sentence 2A. "Ponies like carrots." will be represented as sentence 2B. "I gave them carrots as a reward for riding the horse" will be represented as sentence 2C.

[0023] The explanation will be given using sentence 2B, "Ponies like carrots." The information processing device calculates a sentence vector "SV2B" by accumulating the word vectors of the words included in sentence 2B. The information processing device compares the sentence vector of sentence 2B with the representative vector of the sentence vector dictionary 50, and identifies the cluster ID to which sentence 2B belongs.

[0024] For example, the information processing device calculates the similarity (cosine similarity or distance) between the sentence vector and the representative vector, identifies the representative vector that has the maximum similarity to the sentence vector, and sets the cluster ID of the identified representative vector as the cluster ID to which sentence 2B belongs.

[0025] When the similarity between the sentence vector "SV2B" of sentence 2B and the representative vector of cluster Cr1 is maximum, the information processing device identifies the cluster ID to which sentence 2B belongs as "Cr1."

[0026] The information processing device registers the input text 2 in the DB 141 in units of records. The information processing device associates a cluster ID, a record pointer, and a position pointer, and registers them in the sentence transposition index TS. The record pointer is the position of a record in DB 141 in which a text having a sentence belonging to the corresponding cluster is registered, and indicates the position from the beginning of DB 141. The position pointer is the position of a record in DB 141 in which a sentence belonging to the corresponding cluster is registered, and indicates the relative position from the record pointer.

[0027] For example, if the sentence vector "SV2B" of sentence 2B belongs to cluster Cr1, in the sentence transposition index TS, the start position (position from the start of DB141) of the corresponding record in DB141 in which input text 2 is registered is set to record pointer (1). The position pointer (1) is set to the position of sentence 2B registered in the corresponding record in DB141 (relative position from the start of the corresponding record). If other information has already been registered in record pointer (m) and position pointer (m) corresponding to the same cluster ID, the information processing device registers the above information in record pointer (m+1) and position pointer (m+1). m corresponds to a natural number.

[0028] The information processing device also identifies cluster IDs for sentences 2A and 2C based on the sentence vector in the same manner as for sentence 2B, and registers the cluster ID, record pointer, and position pointer in the sentence transposition index TS in association with each other.

[0029] Even when the information processing device acquires another input text to be registered in the record of the DB 141, the information processing device repeats the above process and updates the sentence transposition index TS.

[0030] As described above, when registering the input text 2 in a record of the DB 141, the information processing device according to the first embodiment uses the sentence vector dictionary 50 to identify which cluster ID the sentence of the input text belongs to, and generates a sentence transposition index TS based on the identification result. In this way, by using the sentence vector dictionary 50 in which the representative vectors and diameters of each cluster of sentences are stored in advance, the sentences can be easily linked to the cluster IDs, thereby reducing the load of the registration process.

[0031] Next, the process of the search phase will be described. Fig. 4 is a diagram for explaining the process of the search phase of the information processing device according to the embodiment 1. When the information processing device receives a search query, it searches for sentences based on the search query and the sentence transposition index TS.

[0032] In FIG. 4, search query 4 is described as "Ponies like carrots." The information processing device calculates a sentence vector "QV4" by integrating the word vectors of the words included in search query 4. The information processing device compares the sentence vector of search query 4 with the representative vector of the sentence vector dictionary 50 to identify the cluster ID to which search query 4 belongs.

[0033] For example, the information processing device calculates the similarity (cosine similarity or distance) between the sentence vector "QV4" of search query 4 and the representative vector. The information processing device identifies the representative vector that has the highest similarity to the sentence vector "QV4" and sets the cluster ID of the identified representative vector as the cluster ID to which search query 4 belongs. Here, the explanation will be given assuming that the cluster ID to which search query 4 belongs is "Cr1."

[0034] The information processing device compares the cluster ID “Cr1” to which the search query 4 belongs with the sentence transposition index TS, and searches DB141 for a sentence corresponding to the search query 4. For example, the information processing device extracts the sentence “Ponies like carrots.”, which is indicated by the record pointer (1) and position pointer (1) set in the row of the cluster ID “Cr1”.

[0035] The information processing device extracts the sentence "Horses like carrots" which is indicated by the record pointer (2) and position pointer (2) set in the row of cluster ID "Cr1". The information processing device extracts the sentence "Hoses likes carrots" which is indicated by the record pointer (3) and position pointer (3) set in the row of cluster ID "Cr1".

[0036] Although the explanation is omitted, the information processing device extracts the sentence indicated by the record pointer (m) and position pointer (m) set in the row of the cluster ID "Cr1".

[0037] As in the above process, the information processing device can efficiently search for sentences corresponding to the search query 4 by using the cluster ID to which the search query 4 belongs and the sentence transposition index TS. In the example described in FIG. 4, the information processing device searches for the sentences "Ponies like carrots," "Horses like carrots," and "Hoses like carrots" corresponding to the search query 4.

[0038] Next, an example of the configuration of the information processing device according to the embodiment 1 will be described. Fig. 5 is a functional block diagram showing the configuration of the information processing device according to the embodiment 1. As shown in Fig. 5, the information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0039] The communication unit 110 is connected to an external device or the like by wire or wirelessly, and transmits and receives information to and from the external device or the like. For example, the communication unit 110 is realized by a NIC (Network Interface Card) or the like. The communication unit 110 may be connected to a network (not shown).

[0040] The input unit 120 is an input device that inputs various types of information to the information processing device 100. The input unit 120 corresponds to a keyboard, a mouse, a touch panel, etc. For example, a user may operate the input unit 120 to input a search query, etc.

[0041] The display unit 130 is a display device that displays information output from the control unit 150. The display unit 130 corresponds to a liquid crystal display, an organic EL (Electro Luminescence) display, a touch panel, etc. For example, search results corresponding to a search query are displayed on the display unit 130.

[0042] The storage unit 140 includes text data 40, a cluster table 41, a word vector dictionary 45, a sentence vector dictionary 50, a DB 141, and a sentence transposition index TS. The storage unit 140 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0043] The text data 40 is information used to generate the sentence vector dictionary 50. For example, the text data 40 includes sentences included in Aozora Bunko and the like, papers published on the network, and the like. The sentences included in the text data 40 correspond to sentences 1 to 4 and the like described in FIG. 1.

[0044] The cluster table 41 holds information about clusters. Fig. 6 is a diagram showing an example of a data structure of the cluster table according to the first embodiment. As shown in Fig. 6, the cluster table 41 associates a cluster ID with cluster data. The cluster ID is information that uniquely identifies a cluster.

[0045] Cluster data includes information on sentence vectors included in a corresponding cluster, etc. FIG. 7 is a diagram showing an example of cluster data. In FIG. 7, cluster data for cluster ID "Cr1" will be described. As shown in FIG. 7, cluster data 5 includes areas 5a and 5b, and tables 5c and 5d. Cluster ID "Cr1" is set in area 5a. The radius of the cluster (cluster Cr1) is set in area 5b.

[0046] Representative vectors (1) to (7) of a cluster (cluster Cr1) are set in table 5c. Sentence vectors belonging to cluster Cr1 are registered in table 5d. Representative vectors (1) to (7) are calculated based on the sentence vectors registered in table 5d. Representative vector (n) is a vector that indicates the center of the n-th component vector of each vector in table 5d.

[0047] Although not shown, it is assumed that the other cluster data shown in FIG. 6 also has information similar to the information explained in FIG.

[0048] Returning to the explanation of FIG. 5, the word vector dictionary 45 is a table that defines codes and word vectors assigned to words. FIG. 8 is a diagram showing an example of the data structure of the word vector dictionary. As shown in FIG. 8, this word vector dictionary 45 has codes, words, and word vectors (1) to (7). A code is a code assigned to a word. A word is a word included in a character string. The word vectors (1) to (7) are vectors assigned to words. The n-th component of a word vector is represented as a word vector (n) (n=1 to 7).

[0049] The sentence vector dictionary 50 corresponds to the sentence vector dictionary 50 described in Fig. 2. As described in Fig. 2, the sentence vector dictionary 50 associates a cluster ID, a representative vector, and a radius.

[0050] The DB 141 registers input text and the like.

[0051] The sentence inversion index TS corresponds to the sentence inversion index TS explained in Figures 3 and 4. The sentence inversion index TS associates a cluster ID with a set of a record pointer and a position pointer. Multiple sets of record pointers and position pointers may be registered for one cluster ID.

[0052] Returning to the description of Fig. 5, the control unit 150 has an acquisition unit 151, a preprocessing unit 152, and a search unit 153. The control unit 150 is realized by, for example, a central processing unit (CPU) or a micro processing unit (MPU). The control unit 150 may also be executed by, for example, an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0053] The acquiring unit 151 acquires various data via the communication unit 110 and the input unit 120. For example, the acquiring unit 151 acquires text data 40 and registers it in the storage unit 140.

[0054] The preprocessing unit 152 generates a sentence inversion index TS based on the text data 40. The preprocessing unit 152 sequentially executes a process of calculating a sentence vector, a process of generating a sentence vector dictionary, and a process of generating a sentence inversion index TS.

[0055] The process of calculating a sentence vector executed by the preprocessing unit 152 will be described. Fig. 9 and Fig. 10 are diagrams for explaining the process of calculating a sentence vector. Fig. 9 will be described. In Fig. 9, a case will be described in which the preprocessing unit 152 extracts sentence 1 "Horses like carrots" included in the text data 40 and calculates a sentence vector of sentence 1.

[0056] The preprocessing unit 152 performs morphological analysis on the sentence 1, "Horses like carrots," to break it down into multiple words. Each broken down word is given a "△ (space)." For example, the sentence 1, "Horses like carrots," is divided into "horse△," "wa△," "carrot△," "ga△," "suki△," "desu△," and ".△."

[0057] The preprocessing unit 152 identifies a code corresponding to each word by comparing each divided word with the word vector dictionary 45, and replaces the code with the word. For example, the words "horse△", "is△", "carrot△", "ga△", "like△", "desu△", and ".△" are replaced with "C1", "C2", "C3", "C4", "C5", "C6", and "C7", respectively.

[0058] We now move on to the description of Fig. 10. The preprocessing unit 152 identifies word vectors (1) to (7) assigned to the codes based on the word vector dictionary 45 and each code. For example, the word vectors (1) to (7) of the code "C1" are wv1-1 to 1-7. The word vectors (1) to (7) of the code "C2" are wv2-1 to 2-7. The word vectors (1) to (7) of the code "C3" are wv3-1 to 3-7.

[0059] The word vectors (1) to (7) of the code "C4" are wv4-1 to 4-7. The word vectors (1) to (7) of the code "C5" are wv5-1 to 5-7. The word vectors (1) to (7) of the code "C6" are wv6-1 to 6-7. The word vectors (1) to (7) of the code "C7" are wv7-1 to 7-7.

[0060] The pre-processing unit 152 calculates the sentence vector SV1 of sentence 1 by integrating the word vectors for each element. For example, the pre-processing unit 152 calculates the first component "SV1-1" of the sentence vector SV1 by integrating wv1-1 to 7-1, which become the respective word vectors (1). The pre-processing unit 152 calculates the second component "SV1-2" of the sentence vector SV1 by integrating wv1-2 to 7-2, which become the respective word vectors (2). The pre-processing unit 152 calculates the third component "SV1-3" of the sentence vector SV1 by integrating wv1-3 to 7-3, which become the respective word vectors (3).

[0061] The pre-processing unit 152 calculates the fourth component "SV1-4" of the sentence vector SV1 by integrating wv1-4 to 7-4, which become the word vectors (4). The pre-processing unit 152 calculates the fifth component "SV1-5" of the sentence vector SV1 by integrating wv1-5 to 7-5, which become the word vectors (5). The pre-processing unit 152 calculates the sixth component "SV1-6" of the sentence vector SV1 by integrating wv1-6 to 7-6, which become the word vectors (6). The pre-processing unit 152 calculates the seventh component "SV1-7" of the sentence vector SV1 by integrating wv1-7 to 7-7, which become the word vectors (7).

[0062] The preprocessing unit 152 repeatedly executes the above process for each sentence included in the text data 40 to calculate the sentence vector for each sentence.

[0063] Next, a process of generating the sentence vector dictionary 50 executed by the preprocessing unit 152 will be described. The process of generating the sentence vector dictionary 50 executed by the preprocessing unit 152 corresponds to the process described in Fig. 1 and Fig. 2. The preprocessing unit 152 classifies each sentence into a plurality of clusters by executing clustering based on the sentence vectors calculated by the above process.

[0064] The preprocessing unit 152 generates cluster data for each cluster, associates the cluster ID with the cluster data, and registers them in the cluster table 41. Here, a case where cluster data is generated for the cluster Cr1 (cluster ID "Cr1") described in Fig. 7 will be described. The preprocessing unit 152 sets sentence vectors belonging to the cluster Cr1 in the feature space in the table 5d.

[0065] The preprocessing unit 152 calculates the center of each component of each sentence vector set in the table 5d to calculate representative vectors (1) to (7) and sets them in the table 5c. The preprocessing unit 152 sets the smallest sphere that contains all sentence vectors belonging to the cluster Cr1 in the feature space. The preprocessing unit 152 registers the diameter of the set sphere in the area 5d.

[0066] The pre-processing unit 152 repeatedly executes the above process for the other clusters, thereby generating cluster data for each cluster and registering the cluster data in the cluster table 41.

[0067] The preprocessing unit 152 extracts the cluster ID, the representative vectors (1) to (7), and the diameter from the cluster table 41, and registers them in the sentence vector dictionary 50.

[0068] Next, a process of generating the sentence inversion index TS executed by the preprocessing unit 152 will be described. The process of generating the sentence inversion index TS by the preprocessing unit 152 corresponds to the process described in Fig. 3. When the preprocessing unit 152 receives the input text 2 to be registered in the DB 141, the preprocessing unit 152 generates the sentence inversion index TS based on each sentence included in the input text 2 and the sentence vector dictionary 50.

[0069] An explanation will be given using sentence 2B in FIG. 3, “Ponies like carrots.” The preprocessing unit 152 calculates a sentence vector “SV2B” by accumulating the word vectors of the words included in sentence 2B. The preprocessing unit 152 calculates the sentence vector “SV2B” of sentence 2B in the same manner as the process of calculating a sentence vector described above. The preprocessing unit 152 compares the sentence vector of sentence 2B with the representative vector in the sentence vector dictionary 50 to identify the cluster ID to which sentence 2B belongs.

[0070] For example, the preprocessing unit 152 calculates the similarity (cosine similarity or distance) between the sentence vector "SV2B" and the representative vector, and identifies the representative vector that has the maximum similarity to the sentence vector "SV2B". The preprocessing unit 152 sets the cluster ID of the identified representative vector as the cluster ID to which sentence 2B belongs.

[0071] When the similarity between the sentence vector "SV2B" of sentence 2B and the representative vector of cluster Cr1 is maximum, the preprocessing unit 152 identifies the cluster ID to which sentence 2B belongs as "Cr1."

[0072] 11 and 12 are diagrams for explaining the process for identifying a cluster ID. In Fig. 11, the feature space 1 includes clusters Cr1, Cr2, and Cr3. For example, as shown in Fig. 11, when the sentence vector SV2B of sentence 2B is included in the area of ​​cluster Cr1, the preprocessing unit 152 identifies the cluster ID to which sentence 2B belongs as "Cr1."

[0073] 12, the feature space 1 includes clusters Cr1, Cr2, and Cr3. For example, as shown in FIG 12, when the sentence vector SV2B of sentence 2B is not included in the areas of the clusters Cr1, Cr2, and Cr3, the preprocessing unit 152 executes the following process. The preprocessing unit 152 calculates the distance l1 between the sentence vector SV2B and the surface of the cluster Cr1, the distance l2 between the sentence vector SV2B and the surface of the cluster Cr2, and the distance l3 between the sentence vector SV2B and the surface of the cluster Cr3.

[0074] The preprocessing unit 152 identifies the cluster ID of the cluster corresponding to the shortest distance among the distances l1 to l3 as the cluster ID to which sentence 2B belongs. For example, when the distance l2 is the shortest distance among the distances l1 to l3, the preprocessing unit 152 identifies the cluster ID to which sentence 2B belongs as "Cr2."

[0075] The preprocessing unit 152 registers the input text 2 in the DB 141. The preprocessing unit 152 associates a cluster ID, a record pointer, and a position pointer, and registers them in the sentence transposition index TS.

[0076] For example, if the sentence vector "SV2B" of sentence 2B belongs to cluster Cr1, the start position of input text 2 registered in DB141 (position from the start of DB141) is set to record pointer (1) in the sentence transposition index TS. The position of sentence 2B registered in DB141 (position relative to the record pointer) is set to position pointer (1). If other information has already been registered in record pointer (m) and position pointer (m) corresponding to the same cluster ID, the preprocessing unit 152 registers the above information in record pointer (m+1) and position pointer (m+1). m corresponds to a natural number.

[0077] The preprocessing unit 152 also identifies cluster IDs for sentences 2A and 2C based on the sentence vector in the same manner as for sentence 2B, and registers the cluster ID, record pointer, and position pointer in the sentence transposition index TS in association with each other.

[0078] The preprocessing unit 152 repeats the above process and updates the sentence transposition index TS even when acquiring other input texts to be registered in the records of the DB 141. The preprocessing unit 152 may also register each sentence of the text data 40 in the records of the DB 141 in the same manner as the sentence 2B, and update the sentence transposition index TS.

[0079] Returning to the description of Fig. 5, when the search unit 153 receives a search query, the search unit 153 searches for a sentence corresponding to the search query from each record of the DB 141 based on the search query and the sentence transposition index TS. The processing of the search unit 153 corresponds to the processing described in Fig. 4.

[0080] The process performed when the search unit 153 acquires search query 4, "Ponies like carrots.", will be described. The search unit 153 calculates a sentence vector "QV4" by integrating the word vectors of the words included in search query 4. The process performed by the search unit 153 to calculate the sentence vector of search query 4 is similar to the process performed by the preprocessing unit 152 to calculate the sentence vector.

[0081] The search unit 153 compares the sentence vector “QV4” of the search query 4 with the representative vectors in the sentence vector dictionary 50 to identify the cluster ID to which the search query 4 belongs.

[0082] For example, the search unit 153 calculates the similarity (cosine similarity or distance) between the sentence vector "QV4" of search query 4 and the representative vector. The search unit 153 identifies the representative vector that has the highest similarity to the sentence vector "QV4", and sets the cluster ID of the identified representative vector as the cluster ID to which search query 4 belongs. Here, the explanation will be given assuming that the cluster ID to which search query 4 belongs is "Cr1".

[0083] The search unit 153 compares the cluster ID "Cr1" to which the search query 4 belongs with the sentence transposition index TS, and searches the DB 141 for a sentence corresponding to the search query 4. For example, the search unit 153 extracts the sentence "Ponies like carrots.", which is indicated by the record pointer (1) and position pointer (1) set in the row of the cluster ID "Cr1".

[0084] The search unit 153 extracts the sentence "Horses like carrots" indicated by the record pointer (2) and position pointer (2) set in the row of the cluster ID "Cr1". The search unit 153 extracts the sentence "Hoses likes carrots" indicated by the record pointer (3) and position pointer (3) set in the row of the cluster ID "Cr1".

[0085] Although the explanation will be omitted, the search unit 153 extracts the sentence indicated by the record pointer (m) and the position pointer (m) set in the row of the cluster ID "Cr1".

[0086] Search unit 153 outputs and displays the search results on display unit 130. Every time search unit 153 receives a search query 4, it repeatedly executes the above process.

[0087] Next, a description will be given of an example of a processing procedure of the information processing device 100. Fig. 13 is a flowchart showing the processing procedure of the preparation process. As shown in Fig. 13, the acquisition unit 151 of the information processing device 100 acquires the text data 40 and registers it in the storage unit 140 (step S101).

[0088] The preprocessing unit 152 of the information processing device 100 calculates a sentence vector of each sentence included in the text data 40 (step S102). The preprocessing unit 152 executes clustering on a plurality of sentence vectors (step S103).

[0089] The preprocessing unit 152 generates a cluster table 41 based on the clustering result (step S104). The preprocessing unit 152 generates a sentence vector dictionary based on the cluster table 41 (step S105).

[0090] The preprocessing unit 152 acquires an input text to be registered in the DB 141 (step S106). The preprocessing unit 152 calculates a sentence vector of the input text (step S107). The preprocessing unit 152 identifies a cluster ID corresponding to a sentence of the input text based on the sentence vector of the input text and the sentence vector dictionary 50 (step S108).

[0091] The preprocessing unit 152 registers the input text in a record of the DB 141 (step S109). The preprocessing unit 152 associates a cluster ID, a record pointer, and a position pointer, and registers them in the sentence transposition index TS (step S110).

[0092] If the pre-processing unit 152 determines to continue the process (Yes in step S111), the process proceeds to step S106. On the other hand, if the pre-processing unit 152 determines not to continue the process (No in step S111), the process ends.

[0093] Fig. 14 is a flowchart showing the procedure of the search process. As shown in Fig. 14, the acquisition unit 151 of the information processing device 100 acquires a search query (step S201).

[0094] The search unit 153 of the information processing device 100 calculates a sentence vector of the search query (step S202). The search unit 153 identifies a cluster ID of the search query based on the sentence vector of the search query and the sentence vector dictionary 50 (step S203).

[0095] The search unit 153 searches for sentences from records in the DB 141 based on the cluster ID of the search query and the sentence transposition index (step S204). The search unit 153 outputs the search result to the display unit 130 (step S205).

[0096] Next, the effect of the information processing device 100 according to the first embodiment will be described. The information processing device 100 calculates sentence vectors of a plurality of sentences included in the text data 40, and executes clustering to generate information on a plurality of clusters, and generates a sentence vector dictionary 50. In the sentence vector dictionary 50, a cluster ID is associated with a representative vector and a radius. For example, when registering a new input text in a record of the DB 141, the information processing device 100 can easily assign a cluster ID to which the sentence vector of the input text belongs by comparing the sentence vector of the input text with the representative vector and radius of the sentence vector dictionary 50.

[0097] When registering the input text 2 in a record of the DB 141, the information processing device 100 uses the sentence vector dictionary 50 to identify which cluster ID the sentence of the input text belongs to, and generates a sentence transposition index TS by associating the identified cluster ID with the stored record and position information of the sentence. In this way, by using the sentence vector dictionary 50, it is possible to easily link a sentence with a cluster ID, thereby reducing the load of the registration process related to the sentence transposition index TS.

[0098] When the information processing device 100 receives a search query, it identifies a cluster ID corresponding to the search query based on the search query and the sentence vector dictionary, and identifies the position of a sentence corresponding to the search query based on the identified cluster ID and the sentence transposition index TS. This makes it possible to efficiently search for a sentence corresponding to the search query. For example, when multiple pieces of position information (record pointers, position pointers) are associated with a cluster ID, it is possible to efficiently search for a sentence corresponding to the search query by identifying multiple pieces of position information at once. EXAMPLES

[0099] In the above-mentioned first embodiment, clustering is performed at the granularity of sentence vectors to generate sentence transposition indexes, but this is not limited to the above. A transposition index (corresponding to a second word transposition index WS2 described later) may be generated at the granularity of word vectors.

[0100] 15 is a functional block diagram showing a configuration of an information processing device according to Example 2. As shown in FIG. 15, this information processing device 200 includes a communication unit 210, an input unit 220, a display unit 230, a storage unit 240, and a control unit 250.

[0101] The explanations regarding the communication unit 210, the input unit 220, and the display unit 230 are similar to the explanations regarding the communication unit 110, the input unit 120, and the display unit 130 described in the first embodiment.

[0102] The storage unit 240 has text data 40, a cluster table 41, a first word vector dictionary 45A, a second word vector dictionary 45B, a sentence vector dictionary 50, a DB 141, and a sentence transposition index TS. The storage unit 240 has a first word transposition index WS1 and a second word transposition index WS2. The storage unit 240 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk.

[0103] The text data 40 is information used to generate the sentence vector dictionary 50. The description of the text data 40 is the same as that in the first embodiment.

[0104] The cluster table 41 holds information about clusters. The cluster table 41 associates cluster IDs with cluster data. The cluster table 41 has cluster data based on the results of clustering at the granularity of sentence vectors, and cluster data based on the results of clustering at the granularity of word vectors.

[0105] The first word vector dictionary 45A is a table that defines codes and word vectors assigned to words. The first word vector dictionary 45A corresponds to the word vector dictionary 45 described with reference to FIG.

[0106] The second word vector dictionary 45B associates cluster IDs, representative vectors, and diameters based on the results of clustering at the granularity of word vectors.

[0107] The sentence vector dictionary 50 associates a cluster ID, a representative vector, and a radius based on the result of clustering at the granularity of the sentence vector. The description of the sentence vector dictionary 50 is the same as that in the first embodiment.

[0108] In the records of DB141, input text and the like are registered.

[0109] The sentence transposition index TS associates the cluster ID to which the sentence vector belongs with a set of a record pointer and a position pointer. The explanation of the sentence transposition index TS is the same as that of the first embodiment.

[0110] The first word transposition index WS1 associates words with position information (record pointers and position pointers).

[0111] The second word transposition index WS2 associates words with position information (record pointer and position pointer). Fig. 16 is a diagram showing an example of the second word transposition index. The record pointer is a position on a record of DB141 in which a text having a word belonging to the corresponding cluster is registered, and indicates a position from the beginning of DB141. The position pointer is a position on a record of DB141 in which a word belonging to the corresponding cluster is registered, and indicates a relative position from the beginning of the record.

[0112] For example, if the word vector of the word "carrot" belongs to cluster wCr1, in the second word transposition index WS2, the record pointer (1) is set to the start position (position from the start of DB141) of input text 2 registered in the record of DB141. The position pointer (1) is set to the position of the word "carrot" registered in DB141 (relative position from the start of the record).

[0113] Returning to the description of Fig. 15, the control unit 250 includes an acquisition unit 251, a preprocessing unit 252, and a search unit 253. The control unit 250 is realized by, for example, a CPU or an MPU. The control unit 150 may also be executed by, for example, an integrated circuit such as an ASIC or an FPGA.

[0114] The acquisition unit 251 acquires various data via the communication unit 210 and the input unit 220. For example, the acquisition unit 151 acquires text data 40 and registers it in the storage unit 240.

[0115] The preprocessing unit 252 generates a sentence transposition index TS, a first word index WS1, and a second word transposition index WS2 based on the text data. The process in which the preprocessing unit 252 generates the sentence transposition index TS is the same as that in the first embodiment.

[0116] The process of generating the first word transposition index WS1 by the preprocessing unit 252 will be described. The preprocessing unit 252 extracts sentences included in the text data 40 and divides them into a plurality of words by performing morphological analysis. The preprocessing unit 252 associates the words with position information (record pointer and position pointer) corresponding to the words and registers them in the first word transposition index WS1. The position information is based on the text data 40 registered in the DB 141. The preprocessing unit 252 repeatedly executes the above process for each word included in the text data 40.

[0117] The process of generating the second word transposition index WS2 by the preprocessing unit 252 will be described. The preprocessing unit 252 extracts sentences included in the text data 40 and divides them into a plurality of words by performing morphological analysis. The preprocessing unit 252 compares each divided word with the first word vector dictionary 45A to identify a code corresponding to each word and replace the word with the code. The preprocessing unit 252 identifies word vectors (1) to (7) assigned to the codes based on the first word vector dictionary 45A and each code.

[0118] The preprocessing unit 252 executes clustering at the granularity of word vectors, generates cluster data for each cluster, associates the cluster ID with the cluster data, and registers them in the cluster table 41. The preprocessing unit 252 registers in the cluster table 41 the cluster results performed at the granularity of word vectors and the cluster results performed at the granularity of sentence vectors described in the first embodiment, distinguishing between them.

[0119] The cluster data processed at the word vector granularity includes a cluster ID, representative vectors (1)-(7), and a radius. The representative vectors (1)-(7) calculated at the word vector granularity are calculated by calculating the center of each component of each word vector. The pre-processing unit 252 registers the cluster ID, representative vectors (1)-(7), and radius in the second word vector dictionary 45B.

[0120] When the preprocessing unit 252 receives the input text 2 to be registered in the DB 141, the preprocessing unit 252 generates a second word transposition index WS2 based on each word included in the input text 2, the first word vector dictionary 45A, and the second word vector dictionary 45B.

[0121] The preprocessing unit 252 selects a word included in the input text 2, and identifies a word vector by comparing the selected word with a word vector in the first word vector dictionary 45A. The preprocessing unit 252 compares the word vector with the second word vector dictionary 45B to identify a cluster ID to which the word vector belongs.

[0122] The preprocessing unit 252 registers the input text 2 in the DB 141. The preprocessing unit 252 associates the cluster ID to which the word vector belongs, the record pointer, and the position pointer, and registers them in the second word inversion index WS2.

[0123] When the search unit 253 receives a search query, it searches the DB 141 for a sentence or word corresponding to the search query based on the search query, the sentence transposition index TS, and the second word transposition index WS2. When the search query is specified as a "sentence", it executes the same process as the search unit 153 described in the first embodiment.

[0124] When the search query is specified by "word", the search unit 253 identifies a word vector based on the word of the search query and the first word vector dictionary 45A, and identifies a cluster ID to which the word belongs based on the word vector and the second word vector dictionary 45B. The search unit 253 compares the cluster ID to which the word of the search query belongs with the second word transposition index WS2, and searches the DB 141 for a word (or a sentence containing the word) corresponding to the search query. The search unit 253 outputs the search result to the display unit 230 for display.

[0125] Even when the search query is set as a "sentence," the search unit 253 may perform a search by dividing the sentence into a plurality of words and executing the above process for each word.

[0126] Next, the effect of the information processing device 200 according to the second embodiment will be described. The information processing device 200 calculates word vectors of multiple words included in the text data 40, executes clustering, generates information about multiple clusters, and sets cluster IDs to which the words belong in the second word vector dictionary 45B. For example, when registering a new input text in the DB 141, the information processing device 200 can easily assign a cluster ID to which the word vector of the input text belongs by comparing the word vector of the input text with the second word vector dictionary 45B.

[0127] Incidentally, by applying the sentence transposition index TS described in the above embodiment to other processes, it is possible to reduce the calculation costs and the like required for the other processes.

[0128] FIG. 17 is a diagram for explaining another application example of the sentence transposition index TS. For example, when performing machine learning of a learning model M1 that translates Japanese sentences into English sentences, training data T1 is used. The learning model M1 corresponds to a CNN (Convolutional Neural Network), an RNN (Recurrent Neural Network), an autoencoder, etc. The training data T1 is training data that uses Japanese sentences as input data and English sentences as correct answer data. For example, the Japanese sentences and English sentences of the training data T1 may be converted into cluster IDs using the sentence vector dictionary 50 or the second word vector dictionary 45B, and machine learning may be performed using the cluster IDs. As a result, the same cluster ID is assigned to similar sentences (sentences belonging to the same cluster), and the types of sentences are reduced (degenerated), so that the amount of calculations such as SoftMax functions in machine learning can be significantly reduced.

[0129] In the first and second embodiments, text consisting of character strings has been described as an example, but it may be a program, an organic compound, a genome, etc. It may be an instruction / variable (program), a functional group (organic compound), a protein basic structure kmer (genome), etc. that corresponds to a word and has a meaning. It may also be a line (program), a functional group primary structure (organic compound), a protein primary structure (genome), etc. that corresponds to a sentence.

[0130] Next, an example of a hardware configuration of a computer that realizes the same functions as the information processing device 100 (200) shown in the above embodiment will be described. Fig. 18 is a diagram showing an example of a hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0131] 18, computer 300 has CPU 301 for executing various arithmetic processes, input device 302 for receiving data input from a user, and display 303. Computer 300 also has communication device 304 for transmitting and receiving data to and from external devices etc. via a wired or wireless network, and interface device 305. Computer 300 also has RAM 306 for temporarily storing various information, and hard disk device 307. Each of devices 301 to 307 is connected to bus 308.

[0132] The hard disk device 307 includes an acquisition program 307a, a pre-processing program 307b, and a search program 307c. The CPU 301 reads out each of the programs 307a to 307c and loads them in the RAM 306.

[0133] The acquisition program 307a functions as an acquisition process 306a. The pre-processing program 307b functions as a pre-processing process 306b. The search program 307c functions as a search process 306c.

[0134] The acquisition process 306a corresponds to the processing of the acquisition units 151 and 252. The pre-processing process 306b corresponds to the processing of the pre-processing unit 152 (252). The search process 306c corresponds to the processing of the search unit 153 (253).

[0135] It is not necessary that each of the programs 307a to 307c is stored in the hard disk device 307 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 300. Then, the computer 300 may read and execute each of the programs 307a to 307c. [Explanation of symbols]

[0136] 100,200 Information processing equipment 110,210 Communications Department 120,220 Input section 130,230 Display section 140,240 storage section 141 DB 150,250 Control section 151,251 Acquisition Department 152,252 Preprocessing section 153,253 Search Department

Claims

1. receiving a plurality of characters or character strings to be registered in a database, and calculating a plurality of vectors corresponding to the plurality of characters or character strings; Classifying the calculated vectors into a plurality of clusters based on inter-vector distances; calculating a representative vector representing one or more vectors included in each of the plurality of clusters and a distance from the representative vector of each of the plurality of clusters according to a magnitude of the one or more vectors included in each of the plurality of clusters; storing, in a storage unit, dictionary information in which the calculated representative vector and the calculated distance are associated with cluster identification information for identifying each cluster; generating index information that associates cluster identification information corresponding to the plurality of vectors with position information of the plurality of characters or character strings registered in the database; A processing method in which processing is carried out by a computer.

2. The processing method according to claim 1 , further comprising the step of: determining cluster identification information corresponding to the plurality of vectors based on the plurality of vectors and the representative vector and the distance of the dictionary information.

3. The processing method according to claim 1, characterized in that, when a search query is received, the computer further executes a process of identifying cluster identification information corresponding to the search query vector based on the search query vector and the representative vector and distance of the dictionary information, and identifying position information of multiple characters or character strings corresponding to the search query based on the cluster identification information corresponding to the search query vector and the index information.

4. receiving a plurality of characters or character strings to be registered in a database, and calculating a plurality of vectors corresponding to the plurality of characters or character strings; Classifying the calculated vectors into a plurality of clusters based on inter-vector distances; calculating a representative vector representing one or more vectors included in each of the plurality of clusters and a distance from the representative vector of each of the plurality of clusters according to a magnitude of the one or more vectors included in each of the plurality of clusters; storing, in a storage unit, dictionary information in which the calculated representative vector and the calculated distance are associated with cluster identification information for identifying each cluster; generating index information that associates cluster identification information corresponding to the plurality of vectors with position information of the plurality of characters or character strings registered in the database; A processing program that causes a computer to execute a process.

5. receiving a plurality of characters or character strings to be registered in a database, and calculating a plurality of vectors corresponding to the plurality of characters or character strings; Classifying the calculated vectors into a plurality of clusters based on inter-vector distances; calculating a representative vector representing one or more vectors included in each of the plurality of clusters and a distance from the representative vector of each of the plurality of clusters according to a magnitude of the one or more vectors included in each of the plurality of clusters; storing, in a storage unit, dictionary information in which the calculated representative vector and the calculated distance are associated with cluster identification information for identifying each cluster; generating index information that associates cluster identification information corresponding to the plurality of vectors with position information of the plurality of characters or character strings registered in the database; An information processing device having a control unit that executes processing.

Images