Information processing program, information processing method, and information processing device

Abstract

This information processing device classifies the vectors of a plurality of sentences stored in a file, according to vector similarity, and generates an inverted index in which the vector of each sentence is associated with the position of the sentence in the file. Upon receiving a search query including a plurality of sentences, the information processing device identifies a characteristic sentence from among the plurality of sentences included in the search query. The information processing device identifies a plurality of similar vectors, which represent vectors similar to the vector of the characteristic sentence, on the basis of the vector of the characteristic sentence, each vector in the inverted index, and the classification result. The information processing device identifies, for each of the plurality of similar vectors, first transition data representing the transition of vectors at positions before and after the similar vector, on the basis of the similar vectors and the inverted index. The information processing device identifies, from the first transition data for the plurality of similar vectors, transition data that is similar to second transition data representing the transition of the vectors of the sentences preceding and following the characteristic sentence in the search query.

Description

[Technical field]

[0001] The present invention relates to an information processing program and the like. [Background technology]

[0002] A huge amount of text data is registered in a database (DB), and it is required to appropriately search the DB for data similar to the data entered by a user.

[0003] An example of a conventional server that performs data searches will be described below. When a server performs a data search, it searches at a granularity such as a word, sentence, or term to document, but as an example, a server that searches for terms similar to a search query will be described here. For example, a search query is specified at the granularity of a term, and a term includes multiple sentences.

[0004] In the case of term-granularity searches, the server calculates vectors of terms included in the DB using a static dictionary or the like that defines vectors of words, and generates an inverted index that indicates the relationship between the vectors of terms and the positions of the terms in the DB. For example, the server calculates the vector of a term by integrating the vectors of multiple sentences included in the term. The vector of a sentence is calculated by integrating the vectors of multiple words included in the sentence.

[0005] When the server receives a granular search query for a term, it calculates vectors of the terms of the search query in the same way as it calculates vectors of terms in the DB, and compares the vectors of the terms of the search query with the inverted index to identify the positions of terms similar to the search query. The server responds with information on the identified terms as search results. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2011-118689 A [Patent Document 2] International Publication No. 2020 / 213158 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the above-mentioned conventional techniques have a problem in that the search for data similar to a search query is not highly accurate.

[0008] For example, when narrowing down candidates for similar terms based on the vectors of the terms in a search query for each of the vectors of multiple terms set in a term-granularity inverted index, the accuracy of the search decreases because each candidate term is composed of multiple sentences and each sentence vector and its transition are different.

[0009] In one aspect, the present invention aims to provide an information processing program, an information processing method, and an information processing device that are capable of searching for data similar to a search query with high accuracy and efficiency. [Means for solving the problem]

[0010] In the first proposal, the computer executes the following process. The computer classifies vectors of multiple sentences stored in a file into similar vectors, and generates an inverted index that associates the vectors of the sentences with the positions of the sentences in the file. When the computer receives a search query including multiple sentences, the computer identifies a characteristic sentence from the multiple sentences included in the search query. The computer identifies multiple similar vectors that indicate vectors similar to the vectors of the characteristic sentence, based on the vectors of the characteristic sentence, each vector of the inverted index, and the classification result. For the multiple similar vectors, the computer identifies first transition data that indicates the transition of vectors at positions before and after the similar vector, based on the similar vectors and the inverted index. The computer identifies transition data that is similar to second transition data that indicates the transition of vectors of sentences before and after the characteristic sentence in the search query, from the first transition data of the multiple similar vectors. Effect of the Invention

[0011] It is possible to search for data similar to a search query with high accuracy and efficiency. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram (1) for explaining the processing of the information processing device according to the present embodiment. [Diagram 2] FIG. 2 is a diagram (2) for explaining the process of the information processing device according to the present embodiment. [Diagram 3] FIG. 3 is a diagram (3) for explaining the process of the information processing device according to the present embodiment. [Figure 4] FIG. 4 is a functional block diagram illustrating a configuration of an information processing device according to the present embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of a data structure of the dictionary data. [Figure 6] FIG. 6 is a diagram illustrating an example of a data structure of the inverted index table. [Figure 7] FIG. 7 is a diagram illustrating an example of the inverted index Two. [Figure 8] FIG. 8 is a diagram illustrating an example of the inverted index Tpa. [Figure 9] FIG. 9 is a diagram for explaining the process of the search unit according to the present embodiment. [Figure 10] FIG. 10 is a flowchart (1) showing a processing procedure of the information processing device according to the present embodiment. [Figure 11] FIG. 11 is a flowchart (2) showing the processing procedure of the information processing device according to the present embodiment. [Figure 12A] FIG. 12A is a diagram illustrating an example of a process for hashing a first index. [Figure 12B] FIG. 12B is a diagram illustrating an example of a process of hashing a second index. [Figure 13A] FIG. 13A is a diagram illustrating an example of a process for restoring a first index. [Figure 13B]FIG. 13B is a diagram illustrating an example of a process for restoring the second index. [Figure 14] FIG. 14 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing apparatus of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an information processing program, an information processing method, and an information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the embodiments. EXAMPLES

[0014] The processing of the information processing device according to the present embodiment will be described. Figs. 1, 2, and 3 are diagrams for explaining the processing of the information processing device according to the present embodiment. Fig. 1 will be described. The information processing device performs preprocessing using text data 141. The text data 141 includes a plurality of terms (paragraphs) 141-1 and 141-2. Fig. 1 shows only terms 141-1 and 141-2, and the illustration of other terms is omitted.

[0015] Item 141-1 includes a plurality of sentences SE1-1, SE1-2, SE1-3, and SE1-n. Item 141-2 includes a plurality of sentences SE2-1, SE2-2, SE2-3, and SE2-n. A sentence includes a plurality of words.

[0016] The information processing device calculates the vector of each of the sentences SE1-1 to SE1-n and sentences SE2-1 to SE2-n. For example, the information processing device calculates the vector of a sentence by accumulating the vectors of words included in the sentence. As will be described later, the vector of each word is set in the dictionary data. In the following description, the vector of a sentence is referred to as a "sentence vector." Also, the vectors shown in this embodiment correspond to variance vectors.

[0017] The sentence vectors of sentences SE1-1, SE1-2, SE1-3, and SE1-n are defined as SV1-1, SV1-2, SV1-3, and SV1-n. The sentence vectors of sentences SE2-1, SE2-2, SE2-3, and SE2-n are defined as SV2-1, SV2-2, SV2-3, and SV2-n.

[0018] The information processing device calculates sentence vectors for other sentences (not shown) in the same manner.

[0019] The information processing device performs clustering based on the sentence vector of each sentence, and classifies similar sentence vectors into the same cluster. In the example shown in Fig. 1, sentence vectors SV1-2 and SV2-2 are classified into the same cluster. The information processing device associates each sentence vector with an ID that uniquely identifies the cluster to which the sentence vector belongs.

[0020] We now move on to the explanation of Fig. 2. The information processing device generates a transposed index Tse based on the result of Fig. 1. The transposed index Tse is information that defines the relationship between a sentence vector and the offset of the first word of the sentence. An offset is set on the x-axis (horizontal axis) of the transposed index Tse, and a sentence vector is set on the y-axis (vertical axis). The offset indicates the position information of the sentence corresponding to the sentence vector.

[0021] For example, if the flag “1” is set in the area where the row of the sentence vector “SV1-2” of the transposed index Tse intersects with the column of the offset “OF1”, this indicates that the position of the sentence in the sentence vector “SV1-2” is the offset “OF1”.

[0022] The sentence vectors set on the y-axis of the transposed index Tse are assigned "cluster numbers" that uniquely identify the clusters classified by the clustering described in FIG. 1. For example, in the transposed index Tse, the sentence vector "SV1-1" is assigned the cluster number "G10." Sentence vectors with the same cluster number are classified into the same cluster. For example, sentence vectors SV1-2 and SV2-2 are assigned the same cluster number "G11," so sentence vectors SV1-2 and SV2-2 are classified into the same cluster.

[0023] Next, a process performed when the information processing device receives a search query 10 will be described. The search query 10 includes multiple sentences. The information processing device identifies a characteristic sentence from the multiple sentences included in the search query 10. For example, the information processing device identifies a sentence that includes many words that appear infrequently as a characteristic sentence. For convenience of explanation, in the example of FIG. 2, sentence SEq will be described as a characteristic sentence (hereinafter, referred to as characteristic sentence SEq). The information processing device calculates a sentence vector SVq of characteristic sentence SEq by accumulating the words included in characteristic sentence SEq based on dictionary data.

[0024] The information processing device compares the sentence vector SVq of the characteristic sentence SEq with the sentence vector on the y-axis of the transposed index Tse to identify similar sentence vectors. Among the sentence vectors on the y-axis of the transposed index Tse, a sentence vector similar to the sentence vector SVq of the characteristic sentence SEq is referred to as a "similar vector."

[0025] The information processing device sequentially compares the sentence vector SVq of the characteristic sentence SEq with the sentence vector on the y-axis of the transposed index Tse, and when it identifies one similar vector, it identifies other similar vectors using the cluster number associated with the identified similar vector as a key.

[0026] For example, if a sentence vector SVq and a sentence vector SV1-2 are similar, a sentence vector SV2-2 classified into the same cluster as the sentence vector SV1-2 is also identified as a similar vector. In the following description, the sentence vectors SV1-2 and SV2-2 are written as similar vectors SV1-2 and SV2-2. The offset of the similar vector SV1-2 is "OF1", and the offset of the similar vector SV2-2 is "OF2".

[0027] 3. The information processing device generates a transposed index Tse-1 by swapping the relationship between the x-axis and the y-axis of the transposed index Tse. A sentence vector is set on the x-axis (horizontal axis) of the transposed index Tse-1, and an offset is set on the y-axis (vertical axis). The information processing device may generate the transposed index Tse-1 at the stage of generating the transposed index Tse.

[0028] The information processing device calculates sentence vectors of a predetermined number of sentences before and after a characteristic sentence SEq included in the search query 10, and arranges the calculated sentence vectors in the order of the sentences in the search query to generate query transition data 11. The horizontal axis of the query transition data 11 corresponds to the order of the sentences in the search query 10, and the vertical axis corresponds to the magnitude of the sentence vector.

[0029] For example, in the query transition data 11, the sentence vector SV10-1 of the (M-1)th sentence of the search query 10, the vector SV10 of the Mth characteristic sentence, and the sentence vector SV10-1 of the (M+1)th sentence are arranged side by side.

[0030] The information processing device generates transition data 12a by extracting sentence vectors offset within a predetermined range before and after the position "OF1" of the similar vector SV1-2 from the transposed index Tse-1.

[0031] For example, in the transition data 12a, a sentence vector SV1-α corresponding to the offset OF1-1 of the transposed index Tse-1, a sentence vector SV1-2 at the offset OF1, and a sentence vector SV1-β at the offset OF1+1 are arranged.

[0032] The information processing device also extracts sentence vectors offset within a predetermined range before and after the position "OF2" of the similar vector SV2-2 from the transposed index Tse-1 to generate transition data 12b. The horizontal axis of the transition data 12a and 12b corresponds to the offset, and the vertical axis corresponds to the magnitude of the sentence vector.

[0033] For example, in the transition data 12b, a sentence vector SV2-α corresponding to the offset OF2-1 of the transposed index Tse-1, a sentence vector SV1-2 at the offset OF2, and a sentence vector SV2-β at the offset OF2+1 are arranged.

[0034] The information processing device calculates the similarity between the query transition data 11 and the transition data 12a, and the similarity between the query transition data 11 and the transition data 12b. Here, a case will be described in which the similarity between the query transition data 11 and the transition data 12a is equal to or greater than a threshold, and the similarity between the query transition data 11 and the transition data 12b is less than the threshold.

[0035] The information processing device identifies the offset of the sentence in the similarity vector SV1-2 corresponding to the transition data 12b based on the transposed index Tse, and extracts a term including the sentence at the identified position as a search result of the search query 10.

[0036] As described above, the information processing device according to the present embodiment classifies similar sentence vectors into the same cluster and generates a transposed index Tse that associates sentence vectors (cluster numbers) with offsets. Upon receiving a search query 10, the information processing device identifies multiple similar vectors based on the characteristic sentences extracted from the search query 10 and the vectors and cluster numbers of the transposed index Tse.

[0037] The information processing device compares query transition data indicating the transition of sentence vectors of sentences before and after the characteristic sentence with transition data indicating the transition of sentence vectors in a predetermined range based on the offset of the similarity vector generated based on the transposed index Tse-1. The information processing device extracts terms of the search query based on the offset of the similarity vector of the transition data in which the similarity is equal to or greater than a threshold value.

[0038] In this way, the information processing device can efficiently identify similar vectors similar to the characteristic sentence by classifying similar sentence vectors into the same cluster. In addition, the information processing device narrows down the search targets based on the query transition data based on the search query and the transition data based on the similar vector and the transposed index Tse-1, so that it can search for terms similar to the sentence of the search query.

[0039] Next, the configuration of the information processing device according to the present embodiment will be described. Fig. 4 is a functional block diagram showing the configuration of the information processing device according to the present embodiment. As shown in Fig. 4, the information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0040] The communication unit 110 is connected to an external device or the like by wire or wirelessly, and transmits and receives information to and from the external device or the like. For example, the communication unit 110 is realized by a NIC (Network Interface Card) or the like. The communication unit 110 may be connected to a network (not shown).

[0041] The input unit 120 is an input device that inputs various types of information to the information processing device 100. The input unit 120 corresponds to a keyboard, a mouse, a touch panel, etc. For example, a user may operate the input unit 120 to input a search query, etc.

[0042] The display unit 130 is a display device that displays information output from the control unit 150. The display unit 130 corresponds to a liquid crystal display, an organic EL (Electro Luminescence) display, a touch panel, etc. For example, search results corresponding to a search query are displayed on the display unit 130.

[0043] The storage unit 140 has text data 141, compressed data 142, dictionary data D1, an inverted index table 143, and a search query 10. The storage unit 140 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0044] The text data 141 is text data having multiple items as described in Fig. 1. In this embodiment, Japanese text data is shown, but the present invention is not limited to this and may be text data in other languages, and may include source programs, genome sequences, chemical structural formulas of organic compounds, image outline PostScript, and the like.

[0045] The compressed data 142 is data obtained by compressing the text data 141 on a word-by-word basis based on the dictionary data D1. An illustration of the compressed data 142 is omitted. In the compressed data 142, a code that separates an item (a line feed code) and a code that separates a sentence (a period code) may be set.

[0046] The dictionary data D1 is a dictionary that defines the relationship between a predetermined word, a static code, and a vector corresponding to the predetermined word. Fig. 5 is a diagram showing an example of the data structure of the dictionary data. As shown in Fig. 5, this dictionary data D1 associates words, static codes, and vectors.

[0047] The inverted index table 143 has inverted indexes of multiple granularities. Fig. 6 is a diagram showing an example of a data structure of the inverted index table. As shown in Fig. 6, the inverted index table 143 has inverted index Two, inverted index Tse, inverted index Tpa, inverted index Tse-1, and inverted index Tpa-1.

[0048] The transposed index Two is information that defines the relationship between a vector of a word and an offset of the word. Fig. 7 is a diagram showing an example of the transposed index Two. An offset is set on the x-axis of the transposed index Two, and a vector of the word is set on the y-axis. The offset indicates the position from the beginning of the code (corresponding to the word) in the compressed data 142.

[0049] For example, offset “0” corresponds to the position of the first code in compressed data 142. Offset “N” corresponds to the position of the “N+1”th code from the beginning of compressed data 142.

[0050] 7, a flag "1" is set at the intersection of the row of the word vector "va1" and the column of the offset "3". This indicates that the code of the word corresponding to the word vector "va1" is located at offset "3" in the compressed data 142. The initial values ​​of the transposition index Two are all set to 0.

[0051] The transposed index Tse is information that defines the relationship between a sentence vector and a sentence offset. The transposed index Tse corresponds to the transposed index Tse described in FIG. 2. An offset is set on the x-axis of the transposed index Tse, and a sentence vector is set on the y-axis. The offset indicates the position from the beginning of the code in the compressed data 142. An offset is assigned at the sentence granularity, but an offset at the word granularity (the position of the code of the first word in the sentence) may be used instead.

[0052] The sentence vector set on the y-axis of the transposed index Tse is assigned a "cluster number" that uniquely identifies the cluster classified by the clustering described in FIG.

[0053] The transposed index Tpa is information that defines the relationship between the vector of a term and the offset of the first word of multiple words included in the first sentence of the term. FIG. 8 is a diagram showing an example of the transposed index Tpa. An offset is set on the x-axis of the transposed index Tpa, and a term vector is set on the y-axis. The term vector is calculated by accumulating the sentence vectors included in the term. The term vector is denoted as a "term vector." The offset indicates the position from the beginning of the code (the code of the first word of multiple words included in the first sentence of the term) in the compressed data 142.

[0054] The transposed indexes Tse-1 and Tpa-1 are transposed indexes in which the correspondence between the x-axis and the y-axis of the transposed indexes Tse and Tpa is swapped. That is, the x-axis of the transposed index Tse-1 is set to a sentence vector, and the y-axis is set to an offset, whereas the x-axis of the transposed index Tpa-1 is set to a term vector, and the y-axis is set to an offset.

[0055] Returning to the description of Fig. 4, the control unit 150 includes a generating unit 151 and a searching unit 152. The control unit 150 is realized by, for example, a central processing unit (CPU) or a micro processing unit (MPU). The control unit 150 may also be executed by, for example, an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0056] The generating unit 151 is a processing unit that generates the compressed data 142 and the inverted index table 143 based on the text data 141 and the dictionary data D1. For example, the generating unit 151 generates the inverted index Tse, etc., by executing the process described in FIG.

[0057] The generating unit 151 scans the text data 141 to identify multiple terms and multiple sentences included in the terms. For example, the generating unit 151 is a portion between line breaks in the text. The generating unit 151 identifies the sentences included in the terms based on periods, punctuation marks, etc.

[0058] The generation unit 151 selects one term of the text data 141, and selects one sentence from multiple sentences included in the selected term. The generation unit 151 performs morphological analysis on the sentence and divides it into multiple words. The generation unit 151 compares the words of the sentence with the dictionary data D1 to identify codes and vectors corresponding to the words. The generation unit 151 compresses (encodes) the sentence by replacing each word included in the sentence with a code. The generation unit 151 calculates the sentence vector of the sentence by accumulating vectors corresponding to the words of the sentence.

[0059] The generation unit 151 compresses the sentences by repeatedly executing the above process for each sentence included in one term, and calculates the sentence vector of the sentence. The generation unit 151 calculates the vector of the term by integrating the sentence vectors of the sentences included in the term. The generation unit 151 repeatedly executes the above process for each term, and generates compressed data 142.

[0060] When the generation unit 151 compares the words of a sentence with the dictionary data D1 and assigns a vector to the word, the generation unit 151 sets the relationship between the vector of the word and the offset of the code of the word in the transposition index Two. When the generation unit 151 calculates a sentence vector of a sentence, the generation unit 151 identifies the offset of the code of the first word of the sentence, and sets the relationship between the sentence vector and the offset in the transposition index Tse. When the generation unit 151 calculates a term vector of a term, the generation unit 151 identifies the offset of the code of the first word of the sentence at the beginning of the term, and sets the relationship between the term vector and the offset in the transposition index Tpa.

[0061] Furthermore, the generation unit 151 performs clustering on the sentence vectors of each sentence to classify the sentence vectors into a plurality of clusters. For example, the generation unit 151 clusters the sentence vectors using the k-means method or the like. Based on the result of the clustering, the generation unit 151 assigns the same cluster number to sentence vectors classified into the same cluster, and sets the cluster number to the sentence vector of the transposed index Tse.

[0062] For example, as described in Figures 1 and 2, if sentence vector SV1-2 and sentence vector SV2-2 are classified into the same cluster (cluster with cluster number "G11"), the generation unit 151 sets cluster number "G11" to sentence vector SV1-2 and sentence vector SV2-2 on the y-axis of the transposed index Tse.

[0063] The generating unit 151 generates transposed indexes Tse- 1 and Tpa- 1 by switching the relationship between the x-axis and the y-axis of the transposed indexes Tse and Tpa, and stores the generated inverted indexes in the transposed index table 143 .

[0064] The search unit 152 is a processing unit that, when it receives the search query 10, searches for terms similar to the search query 10. The search unit 152 identifies a characteristic sentence from a plurality of sentences included in the search query 10. For example, the search unit 152 holds list information of words with low frequency of occurrence, and identifies, from among the plurality of sentences, a sentence with the highest content rate of words with low frequency of occurrence as a characteristic sentence.

[0065] The search unit 152 divides the characteristic sentence into a plurality of words by morphological analysis. Based on the characteristic sentence and the dictionary data D1, the search unit 152 identifies vectors of words included in the characteristic sentence, and calculates the vector of the characteristic sentence by accumulating the vectors of words included in the characteristic sentence. In the following description, the vector of the characteristic sentence is referred to as the "characteristic sentence vector."

[0066] The search unit 152 compares the characteristic sentence vector with the sentence vector on the y-axis of the transposed index Tse to identify a similar sentence vector (similar vector). The search unit 152 calculates the cosine similarity between the characteristic sentence vector and the sentence vector, and identifies the sentence vector whose cosine similarity is equal to or greater than a threshold as a similar vector.

[0067] When the search unit 152 identifies a first similar vector, it identifies other similar vectors using the cluster number associated with the similar vector as a key. For example, as described in FIG. 3, if a sentence vector (characteristic sentence vector) SVq and a sentence vector SV1-2 are similar, the search unit 152 also identifies a sentence vector SV2-2 classified into the same cluster as the sentence vector SV1-2 as a similar vector.

[0068] When the search unit 152 identifies a similar vector, it executes the following process. As described in Fig. 3, the search unit 152 calculates sentence vectors of a predetermined number of sentences before and after the characteristic sentence SEq included in the search query 10, and arranges the calculated sentence vectors in sentence order to generate the query transition data 11. The process in which the search unit 152 calculates the sentence vectors of the sentences is similar to the process of calculating the characteristic sentence vector from the characteristic sentence.

[0069] 3, the search unit 152 generates transition data 12a by extracting sentence vectors having a predetermined range of offsets before and after the position (offset) "OF1" of the similar vector SV1-2 from the transposed index Tse-1. The search unit 152 generates transition data 12b by extracting sentence vectors having a predetermined range of offsets before and after the position "OF2" of the similar vector SV2-2 from the transposed index Tse-1.

[0070] The search unit 152 calculates the similarity between the query transition data 11 and the transition data 12a, and the similarity between the query transition data 11 and the transition data 12b. Here, a case will be described in which the similarity between the query transition data 11 and the transition data 12a is equal to or greater than a threshold, and the similarity between the query transition data 11 and the transition data 12b is less than the threshold.

[0071] The search unit 152 identifies the offset of a term including the sentence of the similar vector SV1-2 based on the offset (OF1) of the sentence of the similar vector SV1-2 corresponding to the transition data 12a and the transposition index Tpa. For example, the search unit 152 traces forward in the transposition index Tpa of the term based on the same offset as the sentence offset, and identifies the offset of the term whose flag is initially set to "1" as the offset of the term including the sentence of the similar vector SV1-2.

[0072] Fig. 9 is a diagram for explaining the processing of the search unit according to this embodiment. In Fig. 9, the explanation is given using the sentence offset (OF1) of the similar vector SV1-2. In the example shown in Fig. 9, when tracing forward in the transposed index Tpa based on the offset OF1, which is the same as the sentence offset, the offset of the term whose flag is initially set to "1" is the offset "3" corresponding to the term vector "vpa1".

[0073] Based on the offset of the identified term, the search unit 152 obtains compressed term information (the code of each word included in the term) from the compressed data 142. The search unit 152 generates data of the term by decoding the code of each word included in the term based on the dictionary data D1. The search unit 152 outputs the decoded data of the term to the display unit 130 as a search result.

[0074] Incidentally, the search unit 152 may further narrow down the terms to be searched by using the vector of the terms described in Fig. 9. For example, it is assumed that there are multiple pieces of transition data whose similarity is equal to or greater than a threshold value when comparing the query transition data 11 with multiple pieces of transition data. Here, the transition data whose similarity with the query transition data 11 is equal to or greater than a threshold value is set as the first candidate transition data and the second candidate transition data.

[0075] The search unit 152 acquires a term vector (first term vector) of a term including a sentence of a similar vector corresponding to the first candidate transition data based on the offset of the similar vector corresponding to the first candidate transition data and the transposed index Tpa. The search unit 152 acquires a term vector (second term vector) of a term including a sentence of a similar vector corresponding to the second candidate transition data based on the offset of the similar vector corresponding to the second candidate transition data and the transposed index Tpa.

[0076] The search unit 152 calculates a first similarity between the vector of the search query 10 and a first term vector, and a second similarity between the vector of the search query 10 and a second term vector. The vector of the search query 10 is a vector obtained by integrating sentence vectors included in the search query 10. The search unit 152 may output data of a term corresponding to the larger term vector of the first similarity or the second similarity as a search result. Furthermore, if both the first similarity and the second similarity are equal to or greater than a threshold, data of a term corresponding to both term vectors may be output as a search result.

[0077] Next, an example of a processing procedure of the information processing device 100 according to this embodiment will be described. Fig. 10 is a flowchart (1) showing the processing procedure of the information processing device according to this embodiment. As shown in Fig. 10, the generating unit 151 of the information processing device 100 scans the text data 141 to identify a plurality of terms and a plurality of sentences included in the terms (step S101).

[0078] The generating unit 151 selects one term from the text data 141 (step S102). The generating unit 151 calculates a sentence vector for each of a plurality of sentences included in the selected term based on the dictionary data D1, and encodes the words of the sentence (step S103).

[0079] The generating unit 151 calculates a term vector by integrating the sentence vectors of the sentences included in the term (step S104). The generating unit 151 registers the encoded term in the compressed data (step S105).

[0080] The generating unit 151 updates the transposed indexes Two, Tse, and Tpa (step S106). The generating unit 151 determines whether or not all the terms of the text data 141 have been selected (step S107). If all the terms of the text data 141 have not been selected (step S107, No), the generating unit 151 proceeds to step S102.

[0081] When all the terms of the text data 141 have been selected (step S107, Yes), the generating unit 151 exchanges the correspondence between the x-axis and the y-axis of the transposed indexes Tse and Tpa, generates transposed indexes Tse-1 and Tpa-1, and compresses them by hashing in the vertical direction (step S108). An example of the compression process will be described later.

[0082] Fig. 11 is a flowchart (2) showing the processing procedure of the information processing device according to the present embodiment. As shown in Fig. 11, the search unit 152 of the information processing device 100 receives the search query 10 (step S201). The search unit 152 identifies a characteristic sentence from a plurality of sentences included in the search query 10 (step S202).

[0083] The search unit 152 calculates a characteristic sentence vector of the characteristic sentence by using the dictionary data D1 (step S203). The search unit 152 identifies a similar vector based on the characteristic sentence vector, the sentence vector of the transposed index Tse, and the cluster number (step S204).

[0084] The search unit 152 generates query transition data based on the sentence vectors of sentences before and after the characteristic sentence in the search query (step S205). The search unit 152 generates transition data in which sentence vectors of offsets before and after the offset of the similar vector are arranged based on the transposed index Tse-1 (step S206).

[0085] The search unit 152 calculates the similarity between the query transition data and each transition data, and identifies the transition data whose similarity is equal to or greater than a threshold (step S207). The search unit 152 identifies the offset of a term having a sentence corresponding to the transition data whose similarity is equal to or greater than a threshold (step S208). Note that it is also possible to obtain a term vector based on the offset of a similar sentence vector and the transposed index Tpa-1, thereby improving the accuracy of the similarity evaluation.

[0086] The search unit 152 obtains the encoded term from the compressed data 142 based on the identified offset (step S209). The search unit 152 decodes the encoded term based on the dictionary data D1 (step S210). The search unit 152 outputs information on the decoded term as a search result (step S211).

[0087] Next, the effect of the information processing device 100 according to the present embodiment will be described. The information processing device 100 classifies similar sentence vectors into the same cluster, and generates a transposed index Tse in which sentence vectors (cluster numbers) correspond to offsets. When the information processing device 100 receives a search query 10, the information processing device 100 identifies multiple similar vectors based on the characteristic sentences extracted from the search query 10 and the vectors and cluster numbers of the transposed index Tse. In this way, by classifying similar sentence vectors into the same cluster, it is possible to efficiently identify similar vectors that are similar to the characteristic sentences.

[0088] The information processing device 100 acquires the transition data of the search query indicating the transition of the sentence vectors of the sentences before and after the characteristic sentence, and the sentence vectors of the sentences before and after the candidate similar to the characteristic sentence based on the transposed index Tse-1, and generates transition data of the similar candidate summarizing the transition of the sentence vectors. The transition data of the search query is used as a reference to compare the transition of the sentence vectors of each transition data of the similar candidate, and performs a similarity evaluation. The information processing device 100 extracts information of the terms of the search query based on the offset of the similarity vector of the transition data whose similarity is equal to or greater than a threshold value. In this way, since the search target is narrowed down based on the transition data, it is possible to search for terms similar to the sentence of the search query with high accuracy and efficiency. Furthermore, the search query and the search target may include not only text consisting of multiple sentences, but also source programs, genome base sequences, chemical structural formulas of organic compounds, image outline PostScript, and the like.

[0089] Incidentally, the generating unit 151 of the information processing device 100 generates the transposed index Two, the transposed index Tse, the transposed index Tpa, and the transposed index Tse-1, respectively, but the size may be reduced by executing the following hashing process. In the following description, an index that takes an offset on the horizontal axis and a vector on the vertical axis, such as the transposed index Two, the transposed index Tse, and the transposed index Tpa, is referred to as a "first index." Also, an index that takes a vector on the horizontal axis and an offset on the vertical axis, such as the transposed index Tse-1, is referred to as a "second index."

[0090] FIG. 12A is a diagram for explaining an example of processing for hashing the first index. Here, a 32-bit register is assumed, and each bitmap of the first index 140c is hashed based on prime numbers (bases) of 29 and 31, for example. A case will be explained in which hashed bitmaps h11 and h12 are generated from bitmap b1. Bitmap b1 indicates a bitmap obtained by extracting a certain row of the first index 140c. Hashed bitmap h11 is a bitmap hashed with base "29". Hashed bitmap h12 is a bitmap hashed with base "31".

[0091] The generating unit 151 associates the remainder obtained by dividing the position of each bit in the bitmap b1 by one low with the position of the hashed bitmap. When a "1" is set in the position of the bit in the corresponding bitmap b1, the generating unit 151 performs a process of setting a "1" in the associated position of the hashed bitmap.

[0092] An example of a process for generating a hashed bitmap h11 with a base of "29" from a bitmap b1 will be described. First, the generating unit 151 copies information at positions "0 to 28" of the bitmap b1 to the hashed bitmap h11. Next, since the remainder when the bit position "35" of the bitmap b1 is divided by the base "29" is "6", the position "35" of the bitmap b1 is associated with the position "6" of the hashed bitmap h11. Since "1" is set at the position "35" of the bitmap b1, the generating unit 151 sets "1" at the position "6" of the hashed bitmap h11.

[0093] Since the remainder when the bit position "42" of bit map b1 is divided by the low "29" is "13", position "42" of bit map b1 is associated with position "13" of hashed bitmap h11. Since "1" is set to position "42" of bitmap b1, the generation unit 151 sets "1" to position "13" of hashed bitmap h11.

[0094] The generating unit 151 repeatedly executes the above process for positions equal to or greater than position "29" of the bitmap b1, thereby generating a hashed bitmap h11.

[0095] An example of a process for generating a hashed bitmap h12 with a base of "31" from a bitmap b1 will be described. First, the generating unit 151 copies information at positions "0 to 30" of the bitmap b1 to the hashed bitmap h12. Next, since the remainder when the bit position "35" of the bitmap b1 is divided by the base "31" is "4", the position "35" of the bitmap b1 is associated with the position "4" of the hashed bitmap h12. Since "1" is set at the position "35" of the bitmap b1, the generating unit 151 sets "1" at the position "4" of the hashed bitmap h12.

[0096] Since the remainder when the bit position "42" of bit map b1 is divided by the low "31" is "11," position "42" of bit map b1 corresponds to position "11" of hashed bitmap h12. Since "1" is set to position "42" of bitmap b1, the generation unit 151 sets "1" to position "13" of hashed bitmap h12.

[0097] The generating unit 151 repeatedly executes the above process for positions equal to or greater than "31" in the bitmap b1, thereby generating a hashed bitmap h12.

[0098] The generating unit 151 can reduce the amount of data in the first index 140c by compressing each row of the first index 140c using the folding technique described above. The hashed bitmaps of bases "29" and "31" are stored in the storage unit 140 with information on the rows (vectors) of the bitmaps from which they were generated.

[0099] When generating the second index 140d, the generating unit 151 may hash the second index 140d with an adjacent prime number (base) using a bitmap folding technique to reduce the size. Fig. 12B is a diagram for explaining an example of a process of hashing the second index.

[0100] Here, as an example, each bitmap of the second index 140d is hashed based on the prime numbers (bases) 11 and 13. A case will be described in which hashed bitmaps h21 and h22 are generated from bitmap b2. Bitmap b2 indicates a bitmap obtained by extracting a certain row of the second index 140d. Hashed bitmap h21 is a bitmap hashed using base "11". Hashed bitmap h22 is a bitmap hashed using base "13".

[0101] The generating unit 151 associates the remainder obtained by dividing the position of each bit in the bitmap b2 by one low with the position of the hashed bitmap. When a "1" is set in the position of the bit in the corresponding bitmap b2, the generating unit 151 performs a process of setting a "1" in the associated position of the hashed bitmap.

[0102] An example of a process for generating a hashed bitmap h21 with a base of "11" from bitmap b2 will be described. First, the generation unit 151 copies information at positions "0 to 10" of bitmap b2 to the hashed bitmap h21. Next, since the remainder when dividing the bit position "15" of bitmap b2 by the base "11" is "4", position "15" of bitmap b2 is associated with position "4" of hashed bitmap h11. Since "1" is set at position "15" of bitmap b2, the generation unit 151 sets "1" at position "6" of hashed bitmap h21.

[0103] The generating unit 151 repeatedly executes the above process for positions equal to or greater than "15" in the bitmap b2, thereby generating a hashed bitmap h21.

[0104] An example of a process for generating a hashed bitmap h22 with a base of "13" from bitmap b2 will be described. First, the generation unit 151 copies information at positions "0 to 12" of bitmap b2 to the hashed bitmap h22. Next, since the remainder when the bit position "15" of bitmap b2 is divided by the base "13" is "2", position "15" of bitmap b2 is associated with position "2" of hashed bitmap h22. Since "1" is set at position "15" of bitmap b2, the generation unit 151 sets "1" at position "2" of hashed bitmap h22.

[0105] The generating unit 151 repeatedly executes the above process for positions equal to or greater than "15" in the bitmap b2, thereby generating a hashed bitmap h22.

[0106] The generating unit 151 can reduce the amount of data in the second index 140d by compressing each row of the second index 140d using the folding technique described above. The hashed bitmaps of bases "11" and "13" are stored in the storage unit 140 with information about the row (offset) of the bitmap from which they were generated.

[0107] When the first index 140c is hashed using a folding technique, the search unit 152 reads out the hashed bitmap corresponding to the vector, restores it, and then performs a process of identifying the offset of the vector.

[0108] 13A is a diagram for explaining an example of a process for restoring the first index. Here, as an example, a case will be explained in which the search unit 152 restores the bitmap b1 based on the hashed bitmap h11 and the hashed bitmap h12.

[0109] The search unit 152 generates an intermediate bitmap h11' from the hashed bitmap h11 of base "29." The search unit 152 copies the values ​​at positions 0 to 28 of the hashed bitmap h11 to positions 0 to 28 of the intermediate bitmap h11', respectively.

[0110] For values ​​after position 29 in the intermediate bitmap h11', the search unit 152 repeatedly executes a process of copying the values ​​at positions 0 to 28 in the hashed bitmap h11 for each "29." The example shown in Fig. 13A shows an example in which the values ​​at positions 0 to 14 in the hashed bitmap h11 are copied to positions 29 to 43 in the intermediate bitmap h11'.

[0111] The search unit 152 generates an intermediate bitmap h12' from the hashed bitmap h12 of base "31." The search unit 152 copies the values ​​at positions 0 to 30 of the hashed bitmap h12 to positions 0 to 30 of the intermediate bitmap h12', respectively.

[0112] For values ​​after position 31 in the intermediate bitmap h12', the search unit 152 repeatedly executes a process of copying the values ​​of positions 0 to 30 in the hashed bitmap h12 for each "31." The example shown in Fig. 13A illustrates an example in which the values ​​of positions 0 to 12 in the hashed bitmap h12 are copied to positions 31 to 43 in the intermediate bitmap h12'.

[0113] When the search unit 152 generates the intermediate bitmap h11' and the intermediate bitmap h12', the search unit 152 performs an AND operation on the intermediate bitmap h11' and the intermediate bitmap h12' to restore the bitmap b1 before hashing. The search unit 152 can restore the bitmap corresponding to the vector by repeatedly executing the same process on the other hashed bitmaps.

[0114] When the second index 140d is hashed using a folding technique, the search unit 152 reads out the hashed bitmap corresponding to the offset, restores it, and then performs processing to identify the attribute corresponding to the offset.

[0115] 13B is a diagram for explaining an example of a process for restoring the second index. Here, as an example, a case will be explained in which the search unit 152 restores the bitmap b2 based on the hashed bitmap h21 and the hashed bitmap h22.

[0116] The search unit 152 generates an intermediate bitmap h21' from the hashed bitmap h21 of base "11." The search unit 152 copies the values ​​at positions 0 to 10 of the hashed bitmap h21 to positions 0 to 10 of the intermediate bitmap h21', respectively.

[0117] For values ​​after position 11 in the intermediate bitmap h21', the search unit 152 repeatedly executes a process of copying the values ​​of positions 0 to 10 in the hashed bitmap h21 for each "11." The example shown in Fig. 13B shows an example in which the values ​​of positions 0 to 10 in the hashed bitmap h21 are copied to positions 11 to 21 in the intermediate bitmap h21', and the values ​​of positions 0 to 9 in the hashed bitmap h21 are copied to positions 22 to 31 in the intermediate bitmap h21'.

[0118] The search unit 152 generates an intermediate bitmap h22' from the hashed bitmap h22 of base "13." The search unit 152 copies the values ​​at positions 0 to 12 of the hashed bitmap h22 to positions 0 to 12 of the intermediate bitmap h22', respectively.

[0119] For values ​​after position 13 in the intermediate bitmap h22', the search unit 152 repeatedly executes a process of copying the values ​​of positions 0 to 12 in the hashed bitmap h22 for each "13." The example shown in Fig. 13B shows an example in which the values ​​of positions 0 to 12 in the hashed bitmap h22 are copied to positions 13 to 25 in the intermediate bitmap h22', and the values ​​of positions 0 to 5 in the hashed bitmap h22 are copied to positions 26 to 31 in the intermediate bitmap h22'.

[0120] After generating the intermediate bitmap h21' and the intermediate bitmap h22', the search unit 152 performs an AND operation on the intermediate bitmap h21' and the intermediate bitmap h22' to restore the bitmap b2 before hashing. The search unit 152 can restore the bitmap corresponding to the offset by repeatedly executing the same process on the other hashed bitmaps.

[0121] Next, an example of a hardware configuration of a computer that realizes the same functions as the information processing device 100 described in the above embodiment will be described. Fig. 14 is a diagram showing an example of a hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0122] 14, the computer 200 has a CPU 201 that executes various arithmetic processes, an input device 202 that accepts data input from a user, and a display 203. The computer 200 also has a communication device 204 that transmits and receives data to and from external devices, etc., via a wired or wireless network, and an interface device 205. The computer 200 also has a RAM 206 that temporarily stores various information, and a hard disk device 207. The devices 201 to 207 are connected to a bus 208.

[0123] The hard disk device 207 includes a generation program 207a and a search program 207b. The CPU 201 reads out the programs 207a and 207b and develops them in the RAM 206.

[0124] The generation program 207a functions as the generation process 206a, and the search program 207b functions as the search process 206b.

[0125] The processing of the generation process 206a corresponds to the processing of the generation unit 151. The processing of the search process 206b corresponds to the processing of the search unit 152.

[0126] It should be noted that each of the programs 207a and 207b does not necessarily have to be stored in the hard disk device 207 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a and 207b. [Explanation of symbols]

[0127] 10 Search Queries 100 Information processing device 110 Communications Department 120 Input section 130 Display section 140 Storage section 141 Text Data 142 Compressed Data 143 Inverted Index Table 150 Control section 151 Generation part 152 Search Department

Claims

1. On the computer, The vectors of multiple sentences stored in a file are classified into similar vectors, generating an inverted index that associates the sentence vector with the position of the sentence in the file; When a search query including multiple sentences is received, a characteristic sentence is identified from the multiple sentences included in the search query; Identifying a plurality of similar vectors indicating vectors similar to the vector of the feature sentence based on the vector of the feature sentence, each vector of the inverted index, and the result of the classification; Identifying first transition data indicating a transition of vectors at positions before and after the similar vector based on the similar vector and the transposed index for the plurality of similar vectors; Identifying transition data similar to second transition data indicating transitions of vectors of sentences before and after the characteristic sentence in the search query from the first transition data of the plurality of similar vectors. An information processing program that causes a process to be executed.

2. 2. The information processing program according to claim 1, further comprising a process of identifying a position of data corresponding to the search query based on a similarity vector corresponding to the first transition data identified by the identification process and the transposed index, and searching for data at the identified position.

3. 2. The information processing program according to claim 1, wherein the inverted index has a sentence vector set on a first horizontal axis and a position set on a second vertical axis, and the generating process further executes a process of updating the inverted index by inputting a relationship between the first axis and the second axis.

4. 4. The information processing program according to claim 3, further comprising a process of compressing the data of the inverted index in a vertical direction.

5. The information processing program according to claim 2, characterized in that the search process further uses a similarity between the vector of the search query and a vector of a term that includes the sentence of the similarity vector to identify the location of data corresponding to the search query.

6. The vectors of multiple sentences stored in a file are classified into similar vectors, generating an inverted index that associates the sentence vector with the position of the sentence in the file; When a search query including multiple sentences is received, a characteristic sentence is identified from the multiple sentences included in the search query; Identifying a plurality of similar vectors indicating vectors similar to the vector of the feature sentence based on the vector of the feature sentence, each vector of the inverted index, and the result of the classification; Identifying first transition data indicating a transition of vectors at positions before and after the similar vector based on the similar vector and the transposed index for the plurality of similar vectors; Identifying transition data similar to second transition data indicating transitions of vectors of sentences before and after the characteristic sentence in the search query from the first transition data of the plurality of similar vectors. An information processing method characterized in that the processing is executed by a computer.

7. The vectors of multiple sentences stored in a file are classified into similar vectors, generating an inverted index that associates the sentence vector with the position of the sentence in the file; When a search query including multiple sentences is received, a characteristic sentence is identified from the multiple sentences included in the search query; Identifying a plurality of similar vectors indicating vectors similar to the vector of the feature sentence based on the vector of the feature sentence, each vector of the inverted index, and the result of the classification; Identifying first transition data indicating a transition of vectors at positions before and after the similar vector based on the similar vector and the transposed index for the plurality of similar vectors; Identifying transition data similar to second transition data indicating transitions of vectors of sentences before and after the characteristic sentence in the search query from the first transition data of the plurality of similar vectors. An information processing device comprising a control unit for executing processing.

Images