Information processing devices, programs, and information processing methods

Abstract

It includes: a retrieval object DB (101) that stores multiple retrieval object statements containing multiple retrieval object tokens, each of which is a minimum unit of meaning; a similarity token table storage unit (110) that stores a similarity token table indicating whether the combination of each retrieval object token of the multiple retrieval object tokens with each retrieval token of the multiple retrieval tokens contained in the retrieval statement is highly similar or not, wherein the multiple retrieval tokens are minimum units of meaning; and a statement similarity calculation unit (111) that calculates the inter-token similarity for combinations that are highly similar in the similarity token table, and sets the inter-token similarity to a predetermined value for combinations that are not highly similar in the similarity token table, thereby calculating the inter-token similarity between the retrieval statement and each retrieval object statement of the multiple retrieval object statements.

Description

Technical Field

[0001] This invention relates to information processing apparatus, programs, and methods. Background Technology

[0002] With the widespread use of personal computers and the internet, the amount of electronic documents available to users has increased dramatically. To discover desired documents from such a massive volume of data, efficient document retrieval technologies are required.

[0003] In document retrieval, it is useful to use computers to process the meaning of natural language by representing the smallest unit of a meaningful character or string, namely a token, with a vector representing its meaning.

[0004] Assigning a vector to a token is the mainstream approach, but this method cannot eliminate the ambiguity of meaning for tokens that have multiple meanings depending on the context. Therefore, methods have been proposed to obtain vectors for tokens that can take context into account.

[0005] In document retrieval, it is necessary to measure the semantic similarity between the search query (the input statement) and the object statement (the search target statement). To measure this high-precision similarity, it is useful to calculate the token similarity between the search query and the object statement.

[0006] For example, Non-Patent Document 1 describes a method for calculating the similarity between statements as follows: For each token x contained in the search query x... i Select the search object statement Y j Each token Y contained jk The token with the highest similarity is used to calculate the token similarity φ(x) between combinations of these i words. i Y jk The value is obtained by averaging.

[0007] Existing technical documents

[0008] Non-patent literature

[0009] Non-patent document 1: Co-authored by Kajiwara Tomoyuki and Komachi Mamoru, "Pingyuan いないテキススを", Natural Language Processing, 25(2), 223-249, 2018 Summary of the Invention

[0010] The problem that the invention aims to solve

[0011] In calculating the similarity between statements, it is necessary to calculate the similarity between all tokens contained in the retrieval query and all combinations of all tokens contained in the retrieval object statement. This results in a huge amount of computation and makes it difficult to put into practical use.

[0012] For example, when assigning a vector representation to a token, all similarities between tokens can be pre-calculated and stored in data such as a lookup table, thus eliminating the need for similarity calculation during retrieval. However, when using vector representations of tokens that take into account their context, the meaning of each token varies depending on the context, making it impossible to pre-calculate the similarities between tokens.

[0013] Therefore, one or more aspects of the present invention aim to reduce the computational load of similarity in document retrieval.

[0014] Methods for solving problems

[0015] An information processing apparatus according to one aspect of the present invention is characterized by comprising: a retrieval object storage unit that stores a plurality of retrieval object statements containing a plurality of retrieval object tokens, wherein the plurality of retrieval object tokens are each the smallest unit having meaning; a similarity determination information storage unit that stores similarity determination information indicating whether the combination of each retrieval object token of the plurality of retrieval object tokens with each retrieval token of the plurality of retrieval tokens contained in the retrieval statement has high or low similarity, wherein the plurality of retrieval tokens are the smallest units having meaning; and a statement similarity calculation unit that calculates the inter-to-total ...

[0016] One aspect of the program of the present invention is characterized by enabling a computer to function as: a retrieval object storage unit storing a plurality of retrieval object statements containing a plurality of retrieval object tokens, each of the plurality of retrieval object tokens being a minimum unit of meaning; a similarity determination information storage unit storing similarity determination information indicating whether the combination of each retrieval object token of the plurality of retrieval object tokens with each retrieval token of the plurality of retrieval object tokens contained in the retrieval statement is highly similar or not, wherein the plurality of retrieval object tokens are minimum units of meaning; and a statement similarity calculation unit calculating the inter-to-total ...

[0017] An information processing method of the present invention calculates the inter-statement similarity between multiple search object statements containing multiple search object tokens and a search statement containing multiple search tokens, wherein the multiple search object tokens are each a minimum unit of meaning. The method is characterized by accepting the input of the search statement, calculating the inter-token similarity for combinations that are indicated as high similarity in similarity determination information, and setting the inter-token similarity to a predetermined value for combinations that are indicated as low similarity in the similarity determination information. This allows for the calculation of the inter-statement similarity between the search statement and each of the multiple search object statements, where the similarity determination information indicates whether each of the multiple search object tokens and each combination of the multiple search tokens is highly or lowly similar.

[0018] Invention Effects

[0019] According to one or more methods of the present invention, the computational load of similarity in document retrieval can be reduced. Attached Figure Description

[0020] Figure 1 This is a block diagram that roughly represents the structure of the document retrieval device as an information processing apparatus in Embodiment 1.

[0021] Figure 2 This is a schematic diagram representing an example of the arrangement of search object tokens.

[0022] Figure 3 This is a schematic diagram representing an example of an arrangement of context-dependent expressions for the retrieved objects.

[0023] Figure 4 This is a schematic diagram representing an example of a search query token arrangement.

[0024] Figure 5 This is a schematic diagram representing an example of a permutation of context-dependent expressions for a retrieval query.

[0025] Figure 6 This is a schematic diagram representing an example of a similar token table.

[0026] Figure 7 It is a block diagram that roughly represents the hardware structure used to implement a document retrieval device.

[0027] Figure 8 This is a flowchart illustrating the processing in the context-dependent expression generation unit for the retrieval object in Implementation 1.

[0028] Figure 9 This is a flowchart representing the processing in the data structure conversion section.

[0029] Figure 10 This is a flowchart illustrating the processing within the tokenizer.

[0030] Figure 11 This is a flowchart representing the processing in the context-dependent expression generation section of the retrieval query.

[0031] Figure 12 This is a flowchart illustrating the processing in the similar token table generation section.

[0032] Figure 13 This is a flowchart illustrating the processing in the statement similarity calculation section.

[0033] Figure 14 This is a flowchart representing the processing in the retrieval results output section.

[0034] Figure 15 This is a block diagram that roughly illustrates the structure of the document retrieval device as an information processing apparatus in Embodiment 2.

[0035] Figure 16 This is a flowchart illustrating the processing in the context-dependent expression generation unit for the retrieval object in Implementation Method 2.

[0036] Figure 17 This is a block diagram that roughly represents the structure of the document retrieval device as an information processing apparatus in Embodiment 3.

[0037] Figure 18 This is a flowchart illustrating the processing in the dimensional reduction section of the retrieved object.

[0038] Figure 19 This is a flowchart representing the processing in the retrieval query dimension reduction section. Detailed Implementation

[0039] Implementation Method 1

[0040] Figure 1 This is a block diagram that schematically illustrates the structure of the document retrieval device 100, which is an information processing device according to Embodiment 1.

[0041] The text retrieval device 100 includes a retrieval object database (hereinafter referred to as retrieval object DB) 101, a retrieval object context dependency expression generation unit 102, an information generation unit 103, a retrieval query input unit 106, a token generator 107, a retrieval query context dependency expression generation unit 108, a similar token table storage unit 110, a sentence similarity calculation unit 111, and a retrieval result output unit 112.

[0042] In addition, the information generation unit 103 includes a data structure conversion unit 104, a search database (hereinafter referred to as a search DB) 105, and a similar token table generation unit 109.

[0043] Search object DB 101 is a search object storage unit that stores search object statements and corresponding search object token arrangements. A search object token arrangement is formed by arranging multiple tokens; a statement is defined by a single search object token arrangement. Furthermore, a token is the smallest unit with meaning and is a character or string. Additionally, the tokens contained in a search object token arrangement are also called search object tokens. Furthermore, it is assumed that search object DB 101 stores multiple search object statements and multiple search object token arrangements corresponding to the multiple search object statements.

[0044] Below, as an example, consider a document retrieval task that retrieves the text corresponding to a given search query. Specifically, for the search query "When and when are summer holidays?", consider the task of retrieving the corresponding text "The holidays are as follows. Summer holidays..." from multiple texts. Here, multiple texts become multiple search object statements.

[0045] In this case, the token arrangement of the retrieved object can also be Figure 2 The two-dimensional arrangement shown. Figure 2 In the example of the search object token arrangement shown, the p-th entry is stored in the p-th row, and the q-th search object token of the p-th entry is stored in the p-th row and q-th column. Here, in Figure 2 In this context, the retrieval object token is a character or string enclosed in quotation marks.

[0046] The retrieval object context dependency expression generation unit 102 obtains a sequence of retrieval object tokens from the retrieval object DB 101. Then, the retrieval object context dependency expression generation unit 102 generates a sequence of retrieval object context dependency expressions by arranging the context dependency expressions (i.e., retrieval object context dependency expressions) of all retrieval object tokens included in the obtained sequence of retrieval object tokens. The generated sequence of retrieval object context dependency expressions is provided to the data structure conversion unit 104 and the statement similarity calculation unit 111. Here, the context dependency expression is a vector, and the retrieval object context dependency expression is a retrieval object vector.

[0047] For example, the retrieval object context dependency expression generation unit 102 is a retrieval object vector generation unit that generates a vector, i.e., a retrieval object vector, corresponding to the meaning of the retrieval object tokens contained in the retrieval object token arrangement. Here, the retrieval object context dependency expression generation unit 102 determines the meaning of the retrieval object tokens based on the context of the retrieval object statement corresponding to the retrieval object token arrangement containing the retrieval object tokens, and generates a retrieval object vector in a manner that represents the determined meaning.

[0048] Specifically, the retrieval object context dependency expression generation unit 102 determines the meaning corresponding to the context for each of the multiple retrieval object tokens included in the retrieval object token arrangement. Furthermore, the retrieval object context dependency expression generation unit 102 can generate a retrieval object context dependency expression arrangement by arranging a multidimensional vector representing the determined meaning according to the arrangement of each of the multiple retrieval object tokens.

[0049] The context-dependent expression permutation of the retrieved object can also be, for example, Figure 3 The two-dimensional arrangement shown. Figure 3 In the context dependency representation of the search objects shown, the p-th row stores the text of the p-th entry, and the p-th row and q-th column stores the text of the p-th entry, which is the vector corresponding to the q-th search object token starting from the beginning, as the context dependency representation.

[0050] Furthermore, regarding the method for determining the context-dependent representation corresponding to the retrieved object token, well-known methods can be used. For example, methods for obtaining vector representations of tokens that take into account the context in which they appear are described in the following literature.

[0051] JacobDevlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", CoRR, abs / 1810.04805, May 24, 2018

[0052] The data structure conversion unit 104 obtains the arrangement of context dependency expressions of the search objects from the context dependency expression generation unit 102. Then, the data structure conversion unit 104 converts the obtained arrangement of context dependency expressions of the search objects into a search data structure. The generated search data structure is stored in the search database 105.

[0053] The search data structure can be chosen from any known data structure depending on the k-nearest neighbor search algorithm used. For example, when using an ANN (Approximate Nearest Neighbor search) as the k-nearest neighbor search algorithm, a kd-tree data structure can be chosen. Furthermore, when using LSH (Locality Sensitive Hashing) as the k-nearest neighbor search algorithm, a hash function-based mapping result can be chosen as the data structure. Here, we will illustrate an example of using an ANN as the k-nearest neighbor search algorithm and a kd-tree data structure as the search data structure.

[0054] In addition, these algorithms have been described in the following literature.

[0055] Shunka Wada, "Theory of recent exploration", research report "Theory of recent exploration", research report "コンピュージョンとイメージメディア," no.

[0056] DB 105 stores the search data structure converted by the data structure conversion unit 104.

[0057] The search query input unit 106 is a search input unit that accepts search queries as search statements. A search query contains multiple tokens. The tokens contained in a search query are also called search tokens.

[0058] For example, the search query input unit 106 accepts questions such as "When does summer vacation start and end?" as search queries.

[0059] The token generator 107 obtains a search query from the search query input unit 106. Then, the token generator 107 is a token determination unit that determines search query tokens from the obtained search query and generates a search query token arrangement by arranging the search query tokens. The generated search query arrangement is provided to the search query context dependency expression generation unit 108. Furthermore, the tokens included in the search query token arrangement are also referred to as search query tokens.

[0060] For example, the token generator 107 uses any known technique such as morpheme analysis to determine the tokens as the smallest meaningful units from the search query, and generates a search query token arrangement by arranging the determined tokens.

[0061] Figure 4 This is a schematic diagram representing an example of a search query token arrangement.

[0062] exist Figure 4In the example shown, the r-th token of the retrieval query is stored in the r-th position of the retrieval query token arrangement.

[0063] The retrieval query context dependency expression generation unit 108 obtains a retrieval query token arrangement from the token generator 107. Then, the retrieval query context dependency expression generation unit 108 generates a retrieval query context dependency expression arrangement that arranges the retrieval query context dependency expressions, which is a context dependency expression for all tokens included in the obtained retrieval query token arrangement, i.e., the retrieval query tokens. The generated retrieval query context dependency expression arrangement is provided to the similarity token table generation unit 109 and the statement similarity calculation unit 111. Here, the retrieval query context dependency expression is a retrieval vector.

[0064] For example, the retrieval query context dependency expression generation unit 108 is a retrieval vector generation unit that generates a vector, i.e., a retrieval vector, corresponding to the meaning of the retrieval token. Here, the retrieval query context dependency expression generation unit 108 determines the meaning of the retrieval token based on the context of the retrieval statement and generates a retrieval vector in a manner that represents the determined meaning.

[0065] Specifically, the retrieval query context dependency expression generation unit 108 determines the meaning corresponding to the context for each of the multiple retrieval query tokens included in the retrieval query token arrangement. Then, the retrieval query context dependency expression generation unit 108 generates a retrieval query context dependency expression arrangement by arranging the multidimensional vectors representing the determined meanings according to the respective arrangements of the multiple retrieval query tokens. Furthermore, the method for determining the context dependency expression corresponding to the retrieval query token is the same as that for the retrieval object context dependency expression described above, and a known method can be used.

[0066] Figure 5 This is a schematic diagram representing an example of a permutation of context-dependent expressions for a retrieval query.

[0067] exist Figure 5 In the example shown, the r-th permutation of the context dependency representations of the retrieval query contains a vector that serves as the context dependency representation corresponding to the r-th token of the retrieval query.

[0068] The similarity token table generation unit 109 obtains the retrieval query context dependency expression arrangement from the retrieval query context dependency expression generation unit 108 and the search data structure from the search database 105. Then, based on the obtained retrieval query context dependency expression arrangement and the search data structure, the similarity token table generation unit 109 generates a similarity token table as similarity determination information, representing whether the similarity is relatively high or relatively low, according to each combination of the retrieval object token and the retrieval query token. The generated similarity token table is stored in the similarity token table storage unit 110.

[0069] For example, the similarity token table generation unit 109 calculates the similarity for all combinations of the search object token and the search query token. Using the calculated similarity, it determines whether the similarity is relatively high or relatively low for all combinations of the search object token and the search query token using a known search method that is more efficient than brute-force search. For example, the similarity token table generation unit 109 uses a k-approximate nearest neighbor search to retrieve k nearby points (k being an integer greater than or equal to 1), and retrieves the k search object tokens with relatively high similarity for a given search query token. Then, the similarity token table generation unit 109 sets the k search object tokens found as tokens with relatively high similarity and sets the remaining search object tokens as tokens with relatively low similarity. Furthermore, the k-approximate nearest neighbor search algorithm can use known techniques such as ANN or LSH.

[0070] Figure 6 This is a schematic diagram representing an example of a similar token table.

[0071] Figure 6 The example shown is a lookup table. When the above search query "Summer vacation is..." is entered, the lookup table indicates whether the similarity of each token contained in the search query is relatively high or low among all search object statements.

[0072] exist Figure 6 In the example shown, rows represent search query tokens, and columns represent search object tokens. "〇" indicates relatively high similarity, and "×" indicates relatively low similarity. For example, in the search query token "summer", the search object tokens "rest day" and "summer" have a relatively high similarity among all tokens contained in the search object statement.

[0073] Here, the k-approximate nearest neighbor search algorithm can be applied in the generation of similar token tables, thus having the advantage of reducing the amount of computation.

[0074] In addition, Figure 6 For ease of explanation, the rows store the retrieval query tokens, and the columns store the retrieval object tokens. Here, the rows store the retrieval context dependency representation (i.e., the retrieval vector) corresponding to the retrieval query tokens, and the columns store the retrieval object context dependency representation (i.e., the retrieval object vector) corresponding to the retrieval object tokens.

[0075] As described above, the data structure conversion unit 104, the search DB 105, and the similar token table generation unit 109 constitute the information generation unit 103, which generates a similar token table as similarity determination information.

[0076] The information generation unit 103 searches for one or more nearby points located near a point represented by a single search vector within the multiple search vectors, from among multiple points represented by multiple search object vectors. It then determines one or more combinations of a search token corresponding to the point represented by that single search vector and one or more search object tokens corresponding to the one or more nearby points as having high similarity, and determines one or more combinations of a search token and one or more search object tokens corresponding to one or more points other than the one or more nearby points as having low similarity, thereby generating a similarity token table. Here, the information generation unit 103 uses a search method that is more efficient than a brute-force search that calculates all distances between the point corresponding to the single search vector and the multiple points corresponding to the multiple search object vectors to search for one or more nearby points.

[0077] The similar token table storage unit 110 is a similarity determination information storage unit that stores similar token tables as similarity determination information.

[0078] The similarity token table indicates whether the individual search object tokens of multiple search object tokens and the combinations of individual search tokens of multiple search tokens have high or low similarity.

[0079] The statement similarity calculation unit 111 obtains the similarity token table from the similarity token table storage unit 110, the search object context dependency expression generation unit 102, and the search query context dependency expression generation unit 108. Then, based on the obtained similarity token table, the search object context dependency expression generation unit 102, and the search query context dependency expression generation unit 108, the statement similarity calculation unit 111 calculates the similarity between the search query and the search object statements, i.e., the statement similarity. The calculated statement similarity is provided to the search result output unit 112.

[0080] Here, the inter-statement similarity calculation unit 111 calculates the inter-token similarity for combinations that are indicated as having high similarity in the similarity token table, and sets the inter-token similarity to a predetermined value for combinations that are indicated as having low similarity in the similarity token table, thereby reducing the computational load when calculating inter-statement similarity. Furthermore, when calculating inter-token similarity, the inter-statement similarity calculation unit 111 considers the shorter the distance between a point represented by a search object vector within multiple search object vectors and a point represented by a search vector within multiple search vectors, resulting in a higher inter-token similarity between the combination of that search object vector and that search vector. Then, for each of the multiple search tokens, the inter-statement similarity calculation unit 111 determines the maximum value of the inter-token similarity among the combinations of multiple search object tokens contained in a single search object statement within multiple search object statements, and calculates the inter-statement similarity between the search statement and that single search object statement based on the average of the determined maximum values.

[0081] The following explains how to calculate the similarity between sentences.

[0082] In calculating the similarity between sentences, any token similarity can be used. For example, the Maximum Alignment method described in Non-Patent Document 1 can be used to calculate the similarity between sentences.

[0083] Here, we will first explain the calculation of inter-statement similarity based on the usual Maximum Alignment method, and then explain the calculation of inter-statement similarity using a faster method in Implementation 1.

[0084] In the calculation of inter-statement similarity based on the usual Maximum Alignment method, for each search query token x contained in the search query x, i In the search object statement Y j The tokens Y of each search object included jk In the selection process, the token with the highest similarity among the selected i = |x| retrieval target tokens is chosen. Then, the token similarity φ(x) among the selected i = |x| retrieval target tokens is calculated. i Y jk The similarity between statements is calculated by averaging the values ​​obtained from the averages.

[0085] In the search query x and the j-th search object statement Y j The similarity between statements is set as s(x, Y). j When the maximum alignment method is used, the calculation of the similarity between statements is formulated as shown in equation (1) below.

[0086]

[0087] Here, x i Y represents the i-th search query token for search query x. jk The statement Y represents the search object. j The k-th retrieval object token, φ(x i Y jk ) indicates the retrieval query token x i With the retrieval object token Y jk The similarity between tokens. For token similarity, the distance between the vector of the retrieved query token and the vector of the retrieved object token is used (e.g., cosine similarity of context-dependent expressions), etc.

[0088] In the Maximum Alignment approach, the similarity between the search query and the statements of each search object is calculated using the above approach.

[0089] This is equivalent to calculating the statement similarity s between the search query and all search object statements, as shown in equation (2) below, and generating the statement similarity S(x, Y) between the search query and each search object statement.

[0090]

[0091] Here, the j-th element of S(x, Y) is the query x and the query object Y. j Similarity between statements.

[0092] Next, we will perform the formula transformation for the Maximum Alignment method described above.

[0093] Now, the retrieval query token x is defined by the following equation (3). i The similarity matrix A(i) is formed by the tokens of all retrieved objects.

[0094]

[0095] Here, the similarity matrix A(i) is a matrix of the type represented by the following equation (4).

[0096]

[0097] Additionally, |Y| represents the total number of search query statements. j | represents the number of search object tokens contained in the j-th search object statement.

[0098] Furthermore, for row l that satisfies equation (5) below, since there is no |Yth row l, ... lThe tokens following the +1 row are the searched objects, so the similarity φ between tokens cannot be calculated. Therefore, zero-padding can be performed to fill the similarity between tokens with 0.

[0099]

[0100] Then, the maximum value of similarity, max, is defined as shown in equation (6) below.

[0101]

[0102] In this case, the similarity S(x, Y) between the retrieval query and each retrieval object statement can be transformed as shown in equation (7) below.

[0103]

[0104] As shown in equation (7), in order to find the similarity S(x, Y) between the search query x and each search object statement Y, it is necessary to find the similarity matrix A(i).

[0105] However, the computational cost of calculating the similarity matrix A(i) is O(|x|Σ). j |Y j Therefore, when the retrieved object statement is large-scale, Σ j |Y j The computational cost is enormous, making it impractical.

[0106] Therefore, the statement similarity calculation unit 111 in Embodiment 1 speeds up the calculation of statement similarity.

[0107] In the Maximum Alignment method before speed-up, for each retrieval object statement, the token similarity values ​​between the retrieval query token and all its retrieval object tokens are compared, and the maximum value is obtained. Thus, as shown in equation (6) above, the retrieval query token x is obtained. i With the search object statement Y j The maximum value of the similarity between tokens, max.

[0108] However, in document retrieval tasks, if the token similarity value is relatively high in the retrieval object statement but relatively low in the entire retrieval object statement, the token similarity is less likely to affect the document similarity.

[0109] Therefore, when the token similarity calculation unit 111 is relatively low among all searched object sentences, it omits the calculation of the token similarity (for example, approximates it to 0), thereby speeding up the calculation of document similarity.

[0110] Specifically, the statement similarity calculation unit 111 approximates the similarity matrix A(i) as shown in equation (8) below.

[0111]

[0112] Wherein, γ(x) i Y jk ) is determined by the following equation (9).

[0113]

[0114] Here, Simset(x) i ) is a retrieval query token x that returns a similar token table. i The row contains a column with a value of "○" for the search object token Y. jk Functions of a set.

[0115] For example, in Figure 6 In the example shown, in retrieving the row containing the query token "summer", Simset(x) is used. i Returns the retrieved object tokens "Rest Day" and "Summer".

[0116] The search result output unit 112 obtains the inter-statement similarity from the inter-statement similarity calculation unit 111 and obtains the search object statements from the search object DB101. Then, the search result output unit 112 sorts the search object statements according to the inter-statement similarity and outputs the sorted search object statements as the search results.

[0117] Regarding sorting, you can simply choose any sorting method based on the similarity between statements, such as ascending or descending order.

[0118] Figure 7 This is a block diagram that roughly represents the hardware structure used to implement the document retrieval device 100.

[0119] like Figure 7 As shown, the document retrieval device 100 can be implemented by a computer 190 having a memory 191, a processor 192, an auxiliary storage device 193, a mouse 194, a keyboard 195, and a display device 196.

[0120] Specifically, the aforementioned retrieval object context dependency expression generation unit 102, data structure conversion unit 104, token generator 107, retrieval query context dependency expression generation unit 108, similar token table generation unit 109, statement similarity calculation unit 111, and retrieval result output unit 112 can be partially or entirely composed of a memory 191 and a processor 192, such as a CPU (Central Processing Unit), that executes the program stored in the memory 191. Such a program can be provided via a network, or it can be provided by recording it on a recording medium. That is, such a program can also be provided as a program product, for example.

[0121] Additionally, the retrieval target DB 101, the search DB 105, and the similar token table storage unit 110 can be implemented using the auxiliary storage device 193 via the processor 192. However, the auxiliary storage device 193 does not necessarily need to exist within the document retrieval device 100; it can also be implemented using an auxiliary storage device located in the cloud via a communication interface not shown. Furthermore, the similar token table storage unit 110 can also be implemented using the memory 191.

[0122] The search input unit 106 can be implemented by the processor 192 using a mouse 194 and a keyboard 195 as input devices and a display device 196. Furthermore, the mouse 194 and keyboard 195 function as input units, and the display device 196 functions as a display unit.

[0123] Figure 8 This is a flowchart representing the processing in the context dependency expression generation unit 102 for the retrieved object.

[0124] First, the retrieval object context dependency expression generation unit 102 obtains the retrieval object token arrangement from the retrieval object DB 101 (S10).

[0125] Next, the search object context dependency expression generation unit 102 determines the meaning of each of the search object tokens included in the obtained search object token arrangement based on the context, and arranges the search object context dependency expressions (i.e., search object vectors) representing the determined meanings according to the obtained search object token arrangement, thereby generating a search object context dependency expression arrangement (S11).

[0126] Next, the context dependency expression generation unit 102 provides the generated context dependency expression arrangement of the search object to the data structure conversion unit 104 and the statement similarity calculation unit 111 (S12).

[0127] Figure 9 This is a flowchart illustrating the processing in the data structure conversion unit 104.

[0128] First, the data structure conversion unit 104 obtains the context dependency expression arrangement of the search object from the context dependency expression generation unit 102 (S20).

[0129] Next, the data structure conversion unit 104 converts the obtained context dependency expression arrangement of the search object into a search data structure, which is used to search for a search object token that has a relatively high similarity to the search query token using a search method that is more efficient than brute-force search (S21).

[0130] Next, the data structure conversion unit 104 provides the converted search data structure to the search database 105 (S22). Furthermore, the search database 105 stores the provided search data structure.

[0131] Figure 10 This is a flowchart illustrating the processing in token generator 107.

[0132] The token generator 107 obtains a search query from the search query input unit 106 (S30).

[0133] Next, the token generator 107 determines the smallest meaningful unit of the search query token from the obtained search query, and arranges the determined search query tokens according to the search query to generate a search query token arrangement (S31).

[0134] Next, the token generator 107 provides the generated retrieval query token arrangement to the retrieval query context dependency expression generation unit 108 (S32).

[0135] Figure 11 This is a flowchart representing the processing in the query context dependency expression generation unit 108.

[0136] First, the retrieval query context dependency expression generation unit 108 obtains the retrieval query token arrangement from the token generator 107 (S40).

[0137] Next, the retrieval query context dependency expression generation unit 108 determines the meaning of each of the retrieval query tokens included in the obtained retrieval query token arrangement based on the context, and arranges the vector (hereinafter also called the retrieval query vector) that represents the determined meaning as the obtained retrieval query token arrangement, thereby generating the retrieval query context dependency expression arrangement (S41).

[0138] Next, the retrieval query context dependency expression generation unit 108 provides the generated retrieval query context dependency expression arrangement to the similarity token table generation unit 109 and the statement similarity calculation unit 111 (S42).

[0139] Figure 12 This is a flowchart illustrating the processing in the similar token table generation unit 109.

[0140] First, the similarity token table generation unit 109 obtains the retrieval query context dependency expression arrangement from the retrieval query context dependency expression generation unit 108 (S50).

[0141] In addition, the similar token table generation unit 109 obtains the search data structure from the search database 105 (S51).

[0142] Next, the similarity token table generation unit 109 uses a search method that is more efficient than brute-force search in the search data structure to search for the most similar search object context dependency expressions from all the search object context dependency expressions contained in the sorted search query context dependency expressions, thereby generating a similarity token table that indicates whether the similarity between each search query context dependency expression and each search object context dependency expression is high or low (S52).

[0143] Next, the similar token table generation unit 109 provides the generated similar token table to the similar token table storage unit 110 (S53) for storage.

[0144] Figure 13 This is a flowchart illustrating the processing in the statement similarity calculation unit 111.

[0145] First, the statement similarity calculation unit 111 obtains the similarity token table from the similarity token table storage unit 110 (S60).

[0146] In addition, the statement similarity calculation unit 111 obtains the retrieval query context dependency expression arrangement from the retrieval query context dependency expression generation unit 108 (S61).

[0147] Then, the statement similarity calculation unit 111 obtains the context dependency expression arrangement of the search object from the context dependency expression generation unit 102 (S62).

[0148] Next, the statement similarity calculation unit 111 calculates the token similarity for combinations of search query tokens and search object tokens that are determined to have high similarity by referring to the similarity token table, and sets the combinations that are determined to have low similarity to a predetermined value (e.g., 0), thereby calculating the statement similarity between the search object statement and the search query (S63).

[0149] Next, the sentence similarity calculation unit 111 provides the calculated sentence similarity to the search result output unit 112 (S64).

[0150] Figure 14 This is a flowchart illustrating the processing in the retrieval result output unit 112.

[0151] First, the search result output unit 112 obtains the inter-statement similarity from the inter-statement similarity calculation unit 111 (S70).

[0152] Next, the search result output unit 112 sorts the search object statements according to the obtained statement similarity, thereby generating search results that at least determine the search object statements with the highest statement similarity (S71). Alternatively, the search result output unit 112 can obtain the search object statements from the search object DB 101.

[0153] Next, the search results output unit 112 displays the generated search results, for example... Figure 7 The display device 196 shown outputs the search result (S72).

[0154] As described above, in Implementation 1, when calculating the similarity between statements, the similarity between tokens that are determined to be not very similar can be set to a predetermined value, thereby reducing the computational load of the similarity between statements.

[0155] Implementation Method 2

[0156] Figure 15 This is a block diagram that schematically illustrates the structure of the document retrieval device 200, which is an information processing device according to Embodiment 2.

[0157] The text retrieval device 200 includes a retrieval object DB 101, a retrieval object context dependency expression generation unit 202, an information generation unit 103, a retrieval query input unit 106, a token generator 107, a retrieval query context dependency expression generation unit 108, a similar token table storage unit 110, a statement similarity calculation unit 111, a retrieval result output unit 112, and an ontology DB 213.

[0158] The retrieval object DB 101, information generation unit 103, retrieval query input unit 106, token generator 107, retrieval query context dependency expression generation unit 108, similar token table generation unit 109, similar token table storage unit 110, inter-statement similarity calculation unit 111, and retrieval result output unit 112 in Embodiment 2 are the same as those in Embodiment 1.

[0159] The ontology DB 213 is a storage unit for storing the meaning relation information of the ontology. The ontology is the meaning relation information representing the meaning relation of the token. In embodiment 2, the ontology represents at least one of the synonym relation and the inclusion relation of the token as the meaning relation.

[0160] Furthermore, ontology DB 213, for example, can be accessed via... Figure 7 The processor 192 shown is implemented using an auxiliary storage device 193.

[0161] The retrieval object context dependency expression generation unit 202 obtains a sequence of retrieval object tokens from the retrieval object DB 101. Then, by referring to the ontology stored in the ontology DB 213, the retrieval object context dependency expression generation unit 202 groups the retrieval object tokens contained in the obtained sequence of retrieval object tokens into groups that can be processed as having the same meaning. For example, the retrieval object context dependency expression generation unit 202 groups retrieval object tokens that represent a synonymous or inclusive relationship in the ontology. Specifically, "vacation" and "rest day" both mean "holiday," in other words, they are synonymous, therefore, the retrieval object context dependency expression generation unit 202 groups them together.

[0162] Then, the retrieval object context dependency expression generation unit 202 assigns a retrieval object context dependency expression to a group, generating a permutation of retrieval object context dependency expressions. In other words, the retrieval object context dependency expression generation unit 202 generates a retrieval object vector that serves as the same retrieval object context dependency expression based on multiple retrieval object tokens that have a defined synonymous or inclusive relationship. For example, the retrieval object context dependency expression generation unit 202 can use any one of the retrieval object context dependency expressions from the retrieval object tokens included in a group as the retrieval object context dependency expression for that group, or it can use a representative value (e.g., the average value) of the retrieval object context dependency expressions of the retrieval object tokens included in a group as the retrieval object context dependency expression for that group.

[0163] Figure 16 This is a flowchart illustrating the processing in the context dependency expression generation unit 202 of the retrieval object in Embodiment 2.

[0164] First, the retrieval object context dependency expression generation unit 202 obtains the retrieval object token arrangement from the retrieval object DB 101 (S80).

[0165] In addition, the object context dependency expression generation unit 202 retrieves the ontology from the ontology DB 213 (S81).

[0166] Next, the search object context dependency expression generation unit 202 determines the meaning of each of the search object tokens included in the obtained search object token arrangement based on the context, and groups them according to the determined meaning with reference to the obtained ontology. It assigns a search object context dependency expression to the search object tokens that belong to the group, and assigns a search object context dependency expression for the determined meaning to the search object tokens that do not belong to the group, thereby generating a search object context dependency expression arrangement (S82).

[0167] Next, the context dependency expression generation unit 202 provides the generated context dependency expression arrangement of the search object to the data structure conversion unit 104 and the statement similarity calculation unit 111 (S83).

[0168] As described above, according to Embodiment 2, by grouping the search target tokens, the number of objects in the similarity token table generation unit 109 that determine whether the search query token and the search target token are highly similar is reduced, thereby alleviating the processing load in the similarity token table generation unit 109.

[0169] Implementation Method 3

[0170] Figure 17 This is a block diagram that roughly illustrates the structure of the document retrieval device 300, which is an information processing device according to Embodiment 3.

[0171] The text retrieval device 300 includes a retrieval object DB 101, a retrieval object context dependency expression generation unit 202, an information generation unit 103, a retrieval query input unit 106, a token generator 107, a retrieval query context dependency expression generation unit 108, a similar token table storage unit 110, a sentence similarity calculation unit 111, a retrieval result output unit 112, an ontology DB 213, a retrieval object dimension reduction unit 314, and a retrieval query dimension reduction unit 315.

[0172] The retrieval object DB 101, information generation unit 103, retrieval query input unit 106, token generator 107, retrieval query context dependency expression generation unit 108, similar token table generation unit 109, similar token table storage unit 110, inter-statement similarity calculation unit 111, and retrieval result output unit 112 in Embodiment 3 are the same as those in Embodiment 1.

[0173] However, in Implementation 3, the retrieval query context dependency expression generation unit 108 provides the retrieval query dimension reduction unit 315 and the statement similarity calculation unit 111 with the retrieval query context dependency expression arrangement.

[0174] Furthermore, the retrieval object context dependency expression generation unit 202 and ontology DB 213 in Embodiment 3 are the same as those in Embodiment 2.

[0175] However, in Implementation 3, the retrieval object context dependency expression generation unit 202 provides the retrieval object dependency expression arrangement to the retrieval object dimension reduction unit 314 and the sentence similarity calculation unit 111.

[0176] The retrieval object dimension reduction unit 314 obtains a permutation of retrieval object context dependency expressions from the retrieval object context dependency expression generation unit 202. Then, the retrieval object dimension reduction unit 314 generates a low-dimensional retrieval object context dependency expression (i.e., a low-dimensional retrieval object vector) by dimensionally compressing all retrieval object context dependency expressions contained in the obtained permutation, and permutes this low-dimensional retrieval object context dependency expression to generate a permutation of dimensionally reduced low-dimensional retrieval object context dependency expressions. The retrieval object dimension reduction unit 314 provides the generated permutation of low-dimensional retrieval object context dependency expressions to the data structure conversion unit 104. Furthermore, any known technique such as principal component analysis can be used for dimension compression.

[0177] Furthermore, in Embodiment 3, the data structure conversion unit 104 converts the low-dimensional search object context dependency expression arrangement into a search data structure. The conversion method is the same as in Embodiment 1.

[0178] The retrieval query dimension reduction unit 315 obtains a permutation of retrieval query context dependency expressions from the retrieval query context dependency expression generation unit 108. Furthermore, the retrieval query dimension reduction unit 315 is a retrieval dimension reduction unit that: generates a low-dimensional retrieval query context dependency expression (i.e., a low-dimensional retrieval vector) after dimension compression of all retrieval query context dependency expressions contained in the obtained permutation of retrieval query context dependency expressions; and permutes this low-dimensional retrieval query context dependency expression to generate a permutation of low-dimensional retrieval query context dependency expressions after dimension reduction. The retrieval query dimension reduction unit 315 provides the generated permutation of low-dimensional retrieval query context dependency expressions to the similarity token table generation unit 109. Additionally, any known technique such as principal component analysis can be used for dimension compression.

[0179] Furthermore, the similarity token table generation unit 109 generates a similarity token table using the low-dimensional search query context dependency expression arrangement obtained from the search query dimension reduction unit 315 and the search data structure obtained from the search database 105. The generation method is the same as in Implementation Method 1.

[0180] As described above, in Embodiment 3, the information generation unit 103 generates a similar token table using the low-dimensional search object context dependency expression arrangement and the low-dimensional search query context dependency expression arrangement generated by the search object dimension reduction unit 314.

[0181] Specifically, the information generation unit 103 searches for one or more points (i.e., one or more nearby points) located near a point represented by a low-dimensional retrieval vector within the multiple points represented by multiple low-dimensional retrieval object vectors. It determines one or more combinations of a retrieval token corresponding to the point represented by that low-dimensional retrieval vector and one or more retrieval object tokens corresponding to the one or more nearby points as having high similarity, and determines one or more combinations of a retrieval token and one or more retrieval object tokens corresponding to one or more points other than the one or more nearby points as having low similarity, thereby generating a similar token table. Here, the information generation unit 103 uses a search method that is more efficient than a brute-force search that calculates all distances between the point corresponding to the low-dimensional retrieval vector and the multiple points corresponding to the multiple low-dimensional retrieval object vectors to search for one or more nearby points.

[0182] The above-described retrieval object dimension reduction unit 314 and retrieval query dimension reduction unit 315 can be partially or entirely provided by... Figure 7 The memory 191 shown is composed of a processor 192 that executes the program stored in the memory 191.

[0183] Figure 18 This is a flowchart illustrating the processing in the object dimension reduction section 314.

[0184] First, the retrieval object dimension reduction unit 314 obtains the retrieval object context dependency expression arrangement from the retrieval object context dependency expression generation unit 202 (S90).

[0185] Next, the retrieval object dimension reduction unit 314 generates a low-dimensional retrieval object context dependency expression arrangement by reducing the dimensions of all retrieval object context dependency expressions contained in the obtained retrieval object context dependency expression arrangement (S91).

[0186] Next, the retrieval object dimension reduction unit 314 provides the low-dimensional retrieval object context dependency expression arrangement to the data structure transformation unit 104 (S92).

[0187] Figure 19 This is a flowchart representing the processing in the retrieval query dimension reduction section 315.

[0188] First, the retrieval query dimension reduction unit 315 obtains the retrieval query context dependency expression arrangement from the retrieval query context dependency expression generation unit 108 (S100).

[0189] Next, the retrieval query dimension reduction unit 315 generates a low-dimensional retrieval query context dependency expression arrangement by reducing the dimensions of all retrieval query context dependency expressions contained in the obtained retrieval query context dependency expression arrangement (S101).

[0190] Next, the retrieval query dimension reduction unit 315 provides the low-dimensional retrieval query context dependency expression arrangement to the similar token table generation unit 109 (S102).

[0191] As described above, according to Embodiment 3, even when the dimensions of the retrieval object context dependency expression and the retrieval query context dependency expression are high, the processing load in the similar token table generation unit 109 can be reduced by reducing these dimensions.

[0192] In the embodiments 1 to 3 described above, the retrieval object DB 101 stores multiple retrieval object statements and multiple retrieval object token arrangements corresponding to these multiple retrieval object statements. However, embodiments 1 to 3 are not limited to such examples. For example, the retrieval object DB 101 may store multiple retrieval object statements, and the retrieval object context dependency expression generation unit 102 may generate the corresponding multiple retrieval object token arrangements using known techniques.

[0193] Furthermore, in the embodiments 1 to 3 described above, the token generator 107 generates a search query token arrangement, but embodiments 1 to 3 are not limited to this example. For example, the search query context dependency expression generation unit 108 may also generate a search query token arrangement based on the search query using known techniques.

[0194] Furthermore, in the embodiments 1 to 3 described above, context-dependent expression generation units 102 and 202 for retrieval objects and context-dependent expression generation unit 108 for retrieval queries generate context-dependent vectors based on tokens. However, embodiments 1 to 3 are not limited to such examples. For instance, vectors corresponding one-to-one with tokens can be generated without relying on context. Even in such cases, according to this embodiment, it is possible to reduce the computational burden of inter-statement similarity by not preparing a lookup table that stores inter-token similarity as a similarity between tokens in advance.

[0195] Implementation 3 is formed by adding a retrieval object dimension reduction unit 314 and a retrieval query dimension reduction unit 315 to implementation 2, but they can also be added to implementation 1.

[0196] Label Explanation

[0197] 100, 200, 300: Document retrieval device; 101: Retrieval object database (DB); 102, 202: Retrieval object context dependency expression generation unit; 103, 303: Information generation unit; 104: Data structure conversion unit; 105: Search database; 106: Retrieval query input unit; 107: Token generator; 108: Retrieval query context dependency expression generation unit; 109: Similarity token table generation unit; 111: Statement similarity calculation unit; 112: Retrieval result output unit; 213: Ontology database; 314: Retrieval object dimension reduction unit; 315: Retrieval query dimension reduction unit.

Claims

1. An information processing device, characterized in that, The information processing device has: The retrieval object storage unit stores multiple retrieval object statements containing multiple retrieval object tokens, each of which is a smallest unit with meaning. The similarity determination information storage unit stores similarity determination information, which indicates whether the combination of each search object token of the plurality of search object tokens and each search token of the plurality of search tokens contained in the search statement has high or low similarity, and the plurality of search tokens is the smallest unit with meaning. as well as The statement similarity calculation unit calculates the token similarity for combinations that are indicated as having high similarity in the similarity determination information, and sets the token similarity to a predetermined value to omit the calculation of the token similarity for combinations that are indicated as having low similarity in the similarity determination information, thereby calculating the statement similarity between the search statement and each of the multiple search object statements.

2. The information processing device according to claim 1, characterized in that, The information processing device also has: The retrieval object vector generation unit generates multiple retrieval object vectors, each of which is a vector corresponding to the meaning of the multiple retrieval object tokens. A retrieval vector generation unit generates multiple retrieval vectors, each of which is a vector corresponding to the meaning of a plurality of retrieval tokens; and The information generation unit generates similarity determination information by searching for one or more nearby points located near a point represented by one of the multiple retrieval object vectors from multiple points represented by the multiple retrieval object vectors, and determining one or more combinations of a retrieval token corresponding to the point represented by the one retrieval vector and one or more retrieval object tokens corresponding to the one or more nearby points as high similarity, and determining one or more combinations of a retrieval token and one or more retrieval object tokens corresponding to one or more points other than the one or more nearby points as low similarity. The information generation unit uses a search method that is more efficient than brute-force search, which calculates all distances between the point corresponding to the one retrieval vector and the points corresponding to the multiple retrieval object vectors, to search for the one or more nearby points.

3. The information processing device according to claim 1, characterized in that, The information processing device also has: The retrieval object vector generation unit generates multiple retrieval object vectors, each of which is a vector corresponding to the meaning of the multiple retrieval object tokens. The object dimension reduction unit reduces the dimension of each of the plurality of object vectors to generate a plurality of low-dimensional object vectors. A retrieval vector generation unit generates multiple retrieval vectors, each of which is a vector corresponding to the meaning of a plurality of retrieval tokens. A retrieval dimension reduction unit reduces the dimension of each of the plurality of retrieval vectors, thereby generating a plurality of low-dimensional retrieval vectors; and The information generation unit generates similarity determination information by searching for one or more nearby points located near a point represented by one of the multiple points represented by the multiple low-dimensional retrieval object vectors. It determines one or more combinations of a retrieval token corresponding to the point represented by the one low-dimensional retrieval vector and one or more retrieval object tokens corresponding to the one or more nearby points as having high similarity, and determines one or more combinations of a retrieval token and one or more retrieval object tokens corresponding to one or more points other than the one or more nearby points as having low similarity. The information generation unit uses a search method that is more efficient than brute-force search, which calculates all distances between the point corresponding to the one low-dimensional retrieval vector and the points corresponding to the multiple low-dimensional retrieval object vectors, to search for the one or more nearby points.

4. The information processing apparatus according to claim 2, characterized in that, The information generation unit searches for one or more nearby points by searching for k nearby points using a k-approximate nearest neighbor search, where k is an integer greater than or equal to 1.

5. The information processing apparatus according to claim 3, characterized in that, The information generation unit searches for one or more nearby points by searching for k nearby points using a k-approximate nearest neighbor search, where k is an integer greater than or equal to 1.

6. The information processing apparatus according to any one of claims 2 to 5, characterized in that, The retrieval object vector generation unit determines the meaning of each of the multiple retrieval object tokens based on the context of each of the multiple retrieval object statements, and generates the multiple retrieval object vectors. The retrieval vector generation unit determines the meaning of each of the multiple retrieval tokens based on the context of the retrieval statement, and generates the multiple retrieval vectors.

7. The information processing apparatus according to claim 6, characterized in that, The retrieval object vector generation unit generates the same retrieval object vector based on multiple retrieval object tokens that have synonymous or inclusive relationships with the determined meaning.

8. The information processing apparatus according to claim 1, characterized in that, The information processing device also has: A retrieval object vector generation unit generates multiple retrieval object vectors, each of which is a vector corresponding to the meaning of a specific retrieval object token; and The retrieval vector generation unit generates multiple retrieval vectors, each of which corresponds to the meaning of a specific retrieval token. When calculating the token similarity, the shorter the distance between a point represented by a retrieval object vector within the plurality of retrieval object vectors and a point represented by a retrieval vector within the plurality of retrieval vectors, the higher the token similarity of the combination of the retrieval object vector and the retrieval vector.

9. The information processing apparatus according to any one of claims 1 to 5, 8, characterized in that, The statement similarity calculation unit determines, for each of the plurality of search tokens, the maximum value of the token similarity among the combinations of search tokens of a search object statement contained in a search object statement, and averages the determined maximum values ​​to calculate the statement similarity between the search statement and the search object statement.

10. The information processing apparatus according to claim 6, characterized in that, The statement similarity calculation unit determines, for each of the plurality of search tokens, the maximum value of the token similarity among the combinations of search tokens of a search object statement contained in a search object statement, and averages the determined maximum values ​​to calculate the statement similarity between the search statement and the search object statement.

11. The information processing apparatus according to claim 7, characterized in that, The statement similarity calculation unit determines, for each of the plurality of search tokens, the maximum value of the token similarity among the combinations of search tokens of a search object statement contained in a search object statement, and averages the determined maximum values ​​to calculate the statement similarity between the search statement and the search object statement.

12. A computer-readable recording medium storing a computer program, said computer program calculating the similarity between multiple statements containing multiple search object tokens and multiple statements containing multiple search object tokens, wherein the multiple search object tokens are each the smallest unit of meaning, and the multiple search tokens are the smallest units of meaning, characterized in that, When the computer program is executed by the processor Accept the input of the search query. For combinations that are indicated as having high similarity in the similarity determination information, the token similarity is calculated. For combinations that are indicated as having low similarity in the similarity determination information, the token similarity is set to a predetermined value to omit the calculation of the token similarity. Thus, the statement similarity between the search statement and each of the multiple search object statements is calculated. The similarity determination information indicates whether each of the multiple search object tokens and each combination of the multiple search tokens has high or low similarity.

13. An information processing method, comprising calculating the similarity between multiple statements containing multiple search object tokens and a search statement containing multiple search tokens, wherein the multiple search object tokens are each the smallest unit of meaning, and the multiple search tokens are the smallest units of meaning, characterized in that... Accept the input of the search query. For combinations that are indicated as having high similarity in the similarity determination information, the token similarity is calculated. For combinations that are indicated as having low similarity in the similarity determination information, the token similarity is set to a predetermined value to omit the calculation of the token similarity. Thus, the statement similarity between the search statement and each of the multiple search object statements is calculated. The similarity determination information indicates whether each of the multiple search object tokens and each combination of the multiple search tokens has high or low similarity.

Images