Dimensional compression method, display method, and information processing device

Abstract

To suppress reduction in retrieval accuracy due to a notation variation with text of a retrieval query.SOLUTION: An information processing device circularly disperses and arranges a plurality of base vectors obtained by decomposing components of vectors for each dimension on the basis of the vectors of plural dimensions corresponding to a character string, calculates a value of the first base vector by integrating a value obtained by performing orthogonal transformation on the second base vector other than the first base vector with respect to the first base vector in the plurality of base vectors, compresses the plural dimensions included in the vector to the dimension corresponding to the first base vector, and sets the value of the compressed dimension to the calculated value of the first base vector.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a specification method and the like. [Background technology]

[0002] In conventional search technologies, when compressing and encoding text such as technical books, the text is lexically analyzed, and an inverted index that associates words with their offsets in the text is generated and used for text search. For example, when a search query (text to be searched) is specified, the inverted index is used to identify offsets corresponding to the words in the search query, and text containing the words in the search query is searched for. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-119714 [Patent Document 2] Japanese Patent Application Publication No. 2018-180789 [Patent Document 3] Japanese Patent Application Laid-Open No. 2006-146355 [Patent Document 4] Japanese Patent Application Laid-Open No. 2002-230021 [Non-patent literature]

[0004] [Non-Patent Document 1] Masajiro Iwasaki, "Release of NGT for fast neighborhood search in high-dimensional vector data",<https: / / techblog.yahoo.co.jp / lab / searchlab / ngt-1.0.0 / > , Retrieved March 12, 2019 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the above-described conventional techniques, there are cases where searches cannot be performed due to variations in the granularity of words and sentences between the text of a technical book and the text of a search query.

[0006] For example, because the inverted index described above associates words with their offsets, it is difficult to search for words that do not match the words in the search query, even if they have the same meaning.

[0007] In one aspect, the present invention aims to provide an identification method, a generation method, a dimension reduction method, a display method, and an information processing device that suppress a decrease in search accuracy caused by spelling variations in the text of a search query. [Means for solving the problem]

[0008] In the first proposal, the computer executes the following process: Based on a multi-dimensional vector corresponding to a character string, the computer distributes a plurality of basis vectors obtained by decomposing the vector into components for each dimension in a circular pattern, calculates the value of a first basis vector by accumulating values ​​obtained by orthogonally transforming second basis vectors other than the first basis vector against the first basis vector, compresses the multiple dimensions included in the vector into dimensions corresponding to the first basis vector, and sets the value of the compressed dimensions to the value of the first basis vector calculated by the calculation process. [Effects of the Invention]

[0009] This can reduce the decrease in search accuracy caused by variations in the text of the search query. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram (1) for explaining the processing of the information processing device according to the present embodiment. [Figure 2] FIG. 2 is a diagram (2) for explaining the processing of the information processing device according to the present embodiment. [Figure 3]FIG. 3 is a functional block diagram showing the configuration of the information processing device according to this embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the data structure of the word vector table. [Figure 5] FIG. 5 is a diagram illustrating an example of the data structure of the dimension reduction table. [Figure 6] FIG. 6 is a diagram illustrating an example of the data structure of a word index. [Figure 7] FIG. 7 is a diagram illustrating an example of the data structure of the synonym index. [Figure 8] FIG. 8 is a diagram illustrating an example of the data structure of the synonymous sentence index. [Figure 9A] FIG. 9A is a diagram for explaining the distributed arrangement of basis vectors. [Figure 9B] FIG. 9B is a diagram for explaining dimensionality reduction. [Figure 10] FIG. 10 is a diagram illustrating an example of processing for hashing an inverted index. [Figure 11] FIG. 11 is a diagram for explaining the dimension restoration. [Figure 12] FIG. 12 is a diagram illustrating the process of restoring a hashed bitmap. [Figure 13] FIG. 13 is a diagram illustrating an example of the graph information. [Figure 14] FIG. 14 is a flowchart (1) showing the processing procedure of the information processing device according to this embodiment. [Figure 15] FIG. 15 is a flowchart (2) showing the processing procedure of the information processing device according to the present embodiment. [Figure 16] FIG. 16 is a diagram showing an example of a plurality of synonym indices generated by the generation processing unit. [Figure 17] FIG. 17 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing device according to this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the identification method, generation method, dimensionality reduction method, display method, and information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments. [Example]

[0012] 1 and 2 are diagrams for explaining the processing of an information processing device according to this embodiment. First, FIG. 1 will be described. As shown in FIG. 1, a dimension reduction unit 150b of the information processing device acquires a word vector table 140a. The word vector table 140a is a table that holds information on the vector of each word. The vector of each word included in the word vector table 140a is a vector calculated in advance using Word2Vec or the like, and is, for example, a 200-dimensional vector.

[0013] The dimension reduction unit 150b generates the dimension-reduced word vector table 140b by reducing the dimension of the vector of each word in the word vector table 140a. The dimension-reduced word vector table 140b is a table that stores information on the vector of each word that has been reduced in dimension. Each word vector included in the dimension-reduced word vector table 140b is a three-dimensional vector.

[0014] The dimension reduction unit 150b reduces each of the 200 vectors a i e i (i=1~200) are distributed evenly in a circle. i " is a basis vector. In the following description, the component-decomposed vector will be referred to as a basis vector. The dimension reduction unit 150b selects one prime basis vector and multiplies that basis vector by values ​​obtained by orthogonally transforming the basis vectors of other dimensions. The dimension reduction unit 150b performs the above process on the three prime basis vectors that are divided by the prime number "3" and distributed, thereby reducing the dimension of a 200-dimensional vector to a three-dimensional vector. For example, the dimension reduction unit 150b reduces the dimension of a 200-dimensional vector to a three-dimensional vector by calculating the values ​​of the prime basis vectors of "1", "67", and "131".

[0015] In this embodiment, a three-dimensional vector is used as an example, but vectors of other dimensions may also be used. By dividing by a prime number "three or more" and selecting a base vector of distributed prime numbers, it is possible to achieve highly accurate dimension restoration, although it is lossy. Note that increasing the dividing prime number improves accuracy but decreases the compression rate. In the following description, a 200-dimensional vector will be referred to as a "vector" and a vector compressed to three dimensions will be referred to as a "compressed vector" as appropriate.

[0016] The generation processing unit 150c of the information processing device receives a plurality of text files 10A. The text files 10A are files containing a plurality of sentences each made up of a plurality of words. The generation processing unit 150c encodes each of the plurality of text files 10A on a word-by-word basis based on the dictionary information 15, thereby generating a plurality of compressed text files 10B.

[0017] When generating the compressed text file 10B based on the text file 10A, the generation processor 150c generates a word index 140c, a synonym index 140d, a synonym sentence index 140e, a sentence vector 140f, and a dynamic dictionary 140g.

[0018] The dictionary information 15 is information (static dictionary) that associates words with codes. The generation processing unit 150c refers to the dictionary information 15, assigns a code to each word in the text file 10A, and compresses it. The generation processing unit 150c assigns a dynamic code to words in the text file 10A that do not exist in the dictionary information 15 or that occur infrequently, and compresses them, and registers the words and the dynamic codes in the dynamic dictionary 140g.

[0019] The word index 140c associates a word code (or word ID) with the position of the code of that word. The position of the code of a word is indicated by an offset in the text compressed file 10B. The offset may be defined in any way in multiple text compressed files 10B. For example, if the offset of the code of the last word in a previous text compressed file is "N," the offset of the code of the first word in the next text compressed file can be "N+1," and so on.

[0020] The synonym index 140d associates the compression vector of a word with the position of the code of the word corresponding to the compression vector. The position of the code of a word is indicated by an offset in the text compression file 10B. Here, the same compression vector is assigned to synonymous words even if the codes of the words are different. For example, if words A1, A2, and A3 are synonyms such as "ringo" (Japanese), "apple" (English), and "pomme" (French), respectively, the compression vectors of words A1, A2, and A3 will have almost the same value.

[0021] The synonymous sentence index 140e associates the compression vector of a sentence with the position of the sentence corresponding to the compression vector. The position of a sentence in the text compression file 10B is the position of the code of the first word among the codes of the words included in the sentence. The generation processing unit 150c calculates the compression vector of the sentence by accumulating the compression vectors of each word included in the sentence and stores the result in the sentence vector table 140f. The generation processing unit 150c calculates the similarity of the compression vectors of each sentence included in the text file 10A and classifies multiple sentences whose similarity is equal to or greater than a threshold into the same group. The generation processing unit 150c identifies each sentence belonging to the same group as synonymous sentences and assigns the same compression vector to them. Note that a three-dimensional compression vector is assigned to each sentence vector. It is also possible to compress multiple sentences collectively by distributing each sentence vector in association with a circle in the order of appearance.

[0022] As described above, the information processing device according to this embodiment generates a dimension-compressed word vector table 140b by dimensionally compressing the word vector table 140a, and when compressing the text file 10A, generates a compressed vector and a synonym index 140d and a synonym sentence index 140e that define the appearance positions of synonyms and synonym sentences corresponding to the compressed vector. The synonym index 140d is information that assigns the same compressed vector to each word belonging to the same synonym and defines the appearance positions of words (synonyms) corresponding to the compressed vector. The synonym sentence index 140e is information that assigns the same compressed vector to each sentence belonging to the same synonym sentence and defines the appearance positions of sentences (synonym sentences) corresponding to the compressed vector. This allows for a reduction in data volume compared to a method that assigns a 200-dimensional vector to each word or sentence.

[0023] Now, let us move on to the description of Fig. 2. When the extraction unit 150d of the information processing device receives the search query 20A, it extracts characteristic words 21 and characteristic sentences 22 based on the dimension-reduced word vector table 140b.

[0024] For example, the extraction unit 150d calculates compressed vectors of multiple sentences included in the search query 20A. First, the extraction unit 150d obtains compressed vectors of multiple words included in one sentence from the dimension-compressed word vector table 140b, and restores the compressed vectors of the obtained words to 200-dimensional vectors.

[0025] The extraction unit 150d distributes each of the 200-dimensional decomposed basis vectors equally in a circular pattern. The extraction unit 150d selects one of the basis vectors other than the three prime numbers "1," "67," and "131" divided by the prime number "3" selected by the dimension reduction unit 150b, and calculates the value of the selected basis vector by accumulating the values ​​obtained by orthogonally transforming the basis vectors of the prime numbers "1," "67," and "131" with the selected basis vector. For example, the extraction unit 150d repeatedly performs the above process on each of the basis vectors corresponding to "2 to 66, 68 to 130, and 132 to 200." By performing the above process, the extraction unit 150d restores the compressed vectors of each word included in the search query 20A to a 200-dimensional vector.

[0026] Next, extraction unit 150d calculates a vector for a sentence by integrating the vectors of multiple words included in one sentence. Extraction unit 150d similarly calculates vectors for other sentences included in search query 20A.

[0027] The extraction unit 150d calculates the vector of the search query 20A by integrating the vectors of multiple sentences included in the search query 20A. In the following description, the vector (200 dimensions) of the search query 20A will be referred to as a "query vector."

[0028] The extraction unit 150d sorts the values ​​of each dimension of the query vector in descending order and identifies the top few dimensions. In the following description, the top few dimensions are referred to as "feature dimensions." The extraction unit 150d extracts, from among the multiple sentences included in the search query 20A, sentences that contain many vector values ​​of the feature dimensions as feature sentences 22. Furthermore, the extraction unit 150d extracts, from among the multiple words included in the search query 20A, words that contain many vector values ​​of the feature dimensions as feature words 21.

[0029] The identification unit 150e compares the compression vector of the characteristic word 21 with the compression vector of the synonym index 140d, and identifies the compression vector of the synonym index 140d whose similarity to the compression vector of the characteristic word 21 is equal to or greater than a threshold. The identification unit 150e searches the multiple text-compressed files 10B for a text-compressed file corresponding to the characteristic word 21, based on the offset corresponding to the identified compression vector, and generates the found text-compressed file as a first candidate list 31.

[0030] The identification unit 150e compares the compressed vector of the characteristic sentence 22 with the compressed vector of the synonym sentence index 140e, and identifies the compressed vector of the synonym sentence index 140e whose similarity to the compressed vector of the characteristic sentence 22 is equal to or greater than a threshold. The identification unit 150e searches the multiple text compressed files 10B for a text compressed file corresponding to the characteristic sentence 22, based on the offset corresponding to the identified compressed vector, and generates the searched text compressed file as a second candidate list 32.

[0031] As described above, when a search query 20A is given, the information processing device identifies the feature dimension of the search query 20A and identifies the feature words 21 and feature sentences 22 that contain many vector values ​​of the feature dimension. The information processing device generates a first candidate list 31 based on the compressed vector of the feature word 21 and the synonym index 140d. The information processing device generates a second candidate list 32 based on the compressed vector of the feature sentence 22 and the synonym sentence index 140e. Because the compressed vectors used in the feature words 21, feature sentences 22, synonym index 140d, and synonym sentence index 140e are three-dimensional vectors, it is possible to detect compressed text files that contain words and sentences similar to the search query 20A while keeping the cost of similarity calculation low.

[0032] Next, an example of the configuration of an information processing device according to this embodiment will be described. Fig. 3 is a functional block diagram showing the configuration of the information processing device according to this embodiment. As shown in Fig. 3, this information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0033] The communication unit 110 is a processing unit that executes data communication with an external device (not shown) via a network or the like. The communication unit 110 corresponds to the communication device. For example, the communication unit 110 may receive information such as a text file 10A, dictionary information 15, and a search query 20A from the external device.

[0034] The input unit 120 is an input device for inputting various types of information to the information processing device 100. The input unit 120 corresponds to a keyboard, a mouse, a touch panel, etc. For example, the user may operate the input unit 120 to input a search query 20A.

[0035] The display unit 130 is a display device that displays various information output from the control unit 150. The display unit 130 corresponds to a liquid crystal display, a touch panel, etc. For example, the display unit 130 displays a first candidate list 31 and a second candidate list 32 identified by the identification unit 150e.

[0036] The storage unit 140 includes a text file 10A, a text compression file 10B, a word vector table 140a, a dimensionally compressed word vector table 140b, a word index 140c, a synonym index 140d, and a synonym sentence index 140e. The storage unit 140 also includes a sentence vector table 140f, a dynamic dictionary 140g, dictionary information 15, a search query 20A, a first candidate list 31, and a second candidate list 32. The storage unit 140 corresponds to a semiconductor memory element such as a RAM (Random Access Memory), a ROM (Read Only Memory), or a flash memory, or a storage device such as an HDD (Hard Disk Drive).

[0037] The text file 10A is information that includes multiple sentences. A sentence is information that includes multiple words. For example, sentences are separated by punctuation marks, periods, etc. In this embodiment, multiple text files 10A are registered in the storage unit 140.

[0038] The text compressed file 10B is information obtained by compressing the text file 10A. For example, the text compressed file 10B is generated by compressing the text file 10A on a word-by-word basis based on the dictionary information 15.

[0039] The word vector table 140a is a table that holds information about vectors for each word. FIG. 4 is a diagram showing an example of the data structure of the word vector table. As shown in FIG. 4, the word vector table 140a associates word IDs with word vectors. The word IDs uniquely identify words. Instead of word IDs, code for words defined in the dictionary information 15 or the like may be used. The vectors are calculated in advance using Word2Vec or the like, and are, for example, 200-dimensional vectors.

[0040] The dimension-reduced word vector table 140b is a table that stores information about the compressed vector of each dimensionally reduced word. FIG. 5 is a diagram showing an example of the data structure of the dimension-reduced word vector table. As shown in FIG. 5, the dimension-reduced word vector table 140b associates word IDs with compressed vectors of the words. Note that word codes may be used instead of word IDs.

[0041] The word index 140c associates word codes (or word IDs) with the positions (offsets) of the word IDs. FIG. 6 is a diagram showing an example of the data structure of a word index. In the word index 140c shown in FIG. 6, the horizontal axis indicates offsets in the text compression file 10B. The vertical axis corresponds to word IDs. For example, a flag "1" is set at the intersection of the row of word ID "A01" and the column of offset "2." This indicates that the code of the word with word ID "A01" is located at offset "2" in the text compression file 10B.

[0042] The offset used in this embodiment is the offset when multiple text compressed files 10B are concatenated in order, and indicates the offset from the first text compressed file 10B. Although not shown in the figure, it is assumed that the offset that marks the break between text compressed files is set in word index 140c. The same applies to the offset of synonym index 140d and the offset of synonymous sentence index 140e, which will be described later.

[0043] The synonym index 140d associates the compressed vector of a word with the position (offset) of the code of the word corresponding to the compressed vector. FIG. 7 is a diagram showing an example of the data structure of a synonym index. In the synonym index 140d shown in FIG. 7, the horizontal axis indicates the offset in the text compressed file 10B. The vertical axis corresponds to the compressed vector of the word. The same compressed vector is assigned to multiple words belonging to the same synonym. For example, a flag "1" is set at the intersection of the row of the synonym compressed vector "W3_Vec1" and the offsets "1" and "6." This indicates that one of the codes of multiple words belonging to the synonym of the compressed vector "W3_Vec1" is located at offsets "1" and "6" in the text compressed file 10B. Note that each dimension of the synonym compressed vector is divided by a certain threshold, so the compressed vector has a certain granularity.

[0044] The synonymous sentence index 140e associates the compression vector of a sentence with the position (offset) of the sentence corresponding to the compression vector. The position of a sentence in the text compression file 10B is the position of the code of the first word among the codes of words included in the sentence. FIG. 8 is a diagram showing an example of the data structure of a synonymous sentence index. In the synonymous sentence index 140e shown in FIG. 8, the horizontal axis indicates the offset in the text compression file 10B. The vertical axis corresponds to the compression vector of the sentence. The same compression vector is assigned to multiple sentences belonging to synonymous sentences with the same meaning. For example, a flag "1" is set at the intersection of the row of the synonymous sentence compression vector "S3_Vec1" and the offsets "3" and "30." This indicates that the code of the first word of one of the multiple sentences belonging to the synonymous sentences of the compression vector "S3_Vec1" is located at offsets "3" and "30" in the text compression file 10B. Note that each dimension of the compression vector of the synonymous sentence is divided by a certain threshold, so the compression vector has a certain granularity.

[0045] The sentence vector table 140f is a table that holds information on the compression vector of a sentence. The dynamic dictionary 140g is information that dynamically associates low-frequency words that appear during compression encoding and words that are not registered in the dictionary information 15 with codes. The dictionary information 15 is information (static dictionary) that associates words with codes.

[0046] The search query 20A includes information about a sentence to be searched for. The search query 20A may be a text file including multiple sentences.

[0047] The first candidate list 31 is a list containing compressed text files 10B detected based on characteristic words 21 extracted from the search query 20A.

[0048] The second candidate list 32 is a list containing compressed text files 10B detected based on characteristic sentences 22 extracted from the search query 20A.

[0049] Returning to the explanation of Fig. 3, the control unit 150 includes a receiving unit 150a, a dimensional compression unit 150b, a generation processing unit 150c, an extraction unit 150d, an identification unit 150e, and a graph generation unit 150f. The control unit 150 can be realized by a CPU, an MPU (Micro Processing Unit), or the like. The control unit 150 can also be realized by hardwired logic such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0050] The reception unit 150a is a processing unit that receives various types of information from the communication unit 110 or the input unit 120. When the reception unit 150a receives a plurality of text files 10A, the reception unit 150a registers the plurality of text files 10A in the storage unit 140. When the reception unit 150a receives a search query 20A, the reception unit 150a registers the search query 20A in the storage unit 140.

[0051] The dimension reduction unit 150b is a processing unit that generates the dimension-reduced word vector table 140b by reducing the dimension of the vectors of each word in the word vector table 140a. FIG. 9A is a diagram for explaining the distribution of the basis vectors. First, the dimension reduction unit 150b reduces the dimension of the 200 basis vectors a i e i (i=1 to 200) are distributed equally in a circle (semicircle). The vector A before decomposition and each decomposed basis vector a i e i The relationship between is defined by equation (1).

[0052]

number

[0053] As shown in FIG. 9A, the dimensionality reduction unit 150b reduces the 200 basis vectors a1e1 to a 200 e 200, the positive (solid line + circular arrow) is distributed in the right semicircle and the negative (dotted line + circular arrow) is distributed in the left semicircle. The angles formed by each basis vector are assumed to be equal. For example, the dimensionality reduction unit 150b distributes the basis vectors a1e1 to a 200 e 200 In this embodiment, as an example, the dimension reduction unit 150b selects the basis vectors of the prime numbers divided by the prime number "3" from the basis vectors a1e1 and a 67 e 67 and the basis vector a 131 e 131 Select and.

[0054] 9B is a diagram for explaining dimensionality reduction. First, the dimensionality reduction unit 150b reduces the remaining basis vectors a2e2 to a1e1. 200 e 200 are orthogonally transformed, and the orthogonally transformed basis vectors a2e2~a 200 e 200 The value of the basis vector a1e1 is calculated by accumulating the values ​​of

[0055] As shown in FIG. 9B, the dimension reduction unit 150b reduces the basis vector a 67 e 67 The remaining basis vectors a1e1 (solid line + arrow), a2e2, a3e3~a 66 e 66 , a 68 e 68 ~a 200 e 200 are orthogonally transformed, and the orthogonally transformed basis vectors a1e1 to a 66 e 66 , a 68 e 68 ~a 200 e 200 By accumulating the values ​​of 67 e 67 Calculate the value of

[0056] The dimension reduction unit 150b reduces the basis vector a 131 e 131 For the remaining basis vectors a1e1~a 130 e 130 , a132 e 132 ~a 200 e 200 are orthogonally transformed, and the orthogonally transformed basis vectors a1e1 to a 130 e 130 , a 132 e 132 ~a 200 e 200 By accumulating the values ​​of 131 e 131 Calculate the value of

[0057] The dimension reduction unit 150b reduces the dimension of a 200-dimensional vector to a compressed vector, and converts each component of the compressed vector into the value of the basis vector a1e1, the value of the basis vector a 67 e 67 The value of the basis vector a 131 e 131 As a result, the 200-dimensional vector can be dimensionally compressed into a 3-dimensional vector divided by the prime number "3". The dimension compression unit 150b may perform dimensional compression using KL expansion or the like. The dimension compression unit 150b performs the above-mentioned dimensional compression for each word in the word vector table 140a, thereby generating the dimension-compressed word vector table 140b.

[0058] The generation processing unit 150c receives multiple text files 10A, performs lexical analysis on the character strings contained in the text files 10A, and divides the character strings into words. The generation processing unit 150c compresses the words contained in the multiple text files 10A word by word based on the dictionary information 15, and generates multiple compressed text files 10B. The generation processing unit 150c compares the words in the text files 10A with the dictionary information 15 and compresses each word into a code. The generation processing unit 150c assigns a dynamic code to words in the text files 10A that do not exist in the dictionary information 15, compresses them, and registers the words and the dynamic codes in the dynamic dictionary 140g.

[0059] Simultaneously with the above-mentioned compression encoding, the generation processing unit 150c generates a word index 140c, a synonym index 140d, a synonym sentence index 140e, and a sentence vector table 140f based on the text file 10A.

[0060] An example of the process by which the generation processing unit 150c generates the "word index 140c" will be described. When the generation processing unit 150c hits a specific word ID (word code) in the process of scanning and compressing words in the text file 10A from the beginning, it identifies the offset from the beginning and sets a flag "1" to the part of the word index 140c where the identified offset intersects with the word ID. The generation processing unit 150c generates the word index 140c by repeatedly executing the above process. The initial value of each part of the word index 140c is set to "0".

[0061] An example of the process by which the generation processor 150c generates the "synonym index 140d" will be described. In the process of scanning and compressing the words in the text file 10A from the beginning, the generation processor 150c acquires a compressed vector corresponding to the word to be compressed from the dimension-compressed word vector table 140b. In the following description, this acquired compressed vector will be referred to as the "target compressed vector" where appropriate.

[0062] The generation processing unit 150c calculates the similarity between the target compressed vector and the compressed vectors of each synonym with a certain granularity in the synonym index 140d, and identifies the compressed vector of the synonym index 140d that has the greatest similarity to the target compressed vector. The generation processing unit 150c sets a flag "1" at the intersection of the row of the identified compressed vector and the column of the word offset of the target compressed vector in the synonym index 140d.

[0063] For example, the generation processing unit 150c calculates the similarity of the compressed vectors based on formula (2). Formula (2) shows a case where the similarity between vector A and vector B is calculated and the similarity of the compressed vectors is evaluated.

[0064]

number

[0065] The generation processing unit 150c repeatedly executes the above process to generate the synonym index 140d. Note that the initial value of each part of the synonym index 140d is set to "0".

[0066] An example of the process by which the generation processing unit 150c generates the "synonymous sentence index 140e" will be described. In the process of scanning and compressing the words of the text file 10A from the beginning, the generation processing unit 150c obtains the compressed vectors of each word (code) from the first word (code) of a sentence to the last word (code) of the sentence from the dimension compressed word vector table 140b, and calculates the compressed vector of one sentence by accumulating each obtained compressed vector. Note that the first word of a sentence is the first word of a piece of text or the word following a punctuation mark. The last word of a sentence is the word before a punctuation mark. In the following description, the calculated compressed vector of a sentence will be referred to as the "target compressed vector" as appropriate.

[0067] The generation processing unit 150c calculates the similarity between the target compressed vector and the compressed vector of each synonymous sentence with a certain granularity in the synonym sentence index 140e, and identifies the compressed vector of the synonym sentence index 140e that has the greatest similarity to the target compressed vector. The generation processing unit 150c calculates the similarity between the target compressed vector and each compressed vector based on equation (2). The generation processing unit 150c sets a flag "1" in the synonym sentence index 140e at the intersection of the row of the identified compressed vector and the column of the offset of the first word of the sentence for the target compressed vector.

[0068] The generation processing unit 150c repeatedly executes the above process to generate the synonymous sentence index 140d. Note that the initial value of each part of the synonymous sentence index 140e is set to "0".

[0069] Incidentally, when generating the word index 140c, the synonym index 140d, and the synonym sentence index 140e, the generation processor 150c may associate the compressed vectors with thresholds of the respective basis vectors of a certain granularity without using equation (2) in order to reduce the amount of calculation. Also, in order to reduce the amount of information, each of the transposed indexes 140c, 140d, and 140e may be hashed.

[0070] Fig. 10 is a diagram illustrating an example of the process of hashing a transposed index. In the example illustrated in Fig. 10, a 32-bit register is assumed, and the bitmap of each row of the word index 140c is hashed based on the prime numbers (bases) "29" and "31." Here, as an example, a case where hashed bitmaps h11 and h12 are generated from bitmap b1 will be described.

[0071] Bitmap b1 indicates a bitmap obtained by extracting a certain row of a word index (for example, word index 140c shown in FIG. 6). Hashed bitmap h11 is a bitmap hashed using base "29". Hashed bitmap h12 is a bitmap hashed using base "31".

[0072] The generation processing unit 150c associates the remainder obtained by dividing the position of each bit in the bitmap b1 by one low with the position of the hashed bitmap. When a "1" is set in the position of the corresponding bit in the bitmap b1, the generation processing unit 150c performs a process of setting a "1" in the associated position of the hashed bitmap.

[0073] An example of a process for generating a hashed bitmap h11 with a base of 29 from a bitmap b1 will be described below. First, the generation processing unit 150c copies the information at positions 0 to 28 of the bitmap b1 to the hashed bitmap h11. Next, since the remainder when the bit position 35 of the bitmap b1 is divided by the base 29 is 6, the position 35 of the bitmap b1 is associated with the position 6 of the hashed bitmap h11. Because a 1 is set at position 35 of the bitmap b1, the generation processing unit 150c sets a 1 at position 6 of the hashed bitmap h11.

[0074] The remainder when the bit position "42" of bitmap b1 is divided by the low "29" is "13," so position "42" of bitmap b1 corresponds to position "13" of hashed bitmap h11. Because "1" is set at position "42" of bitmap b1, the generation processing unit 150c sets "1" at position "13" of hashed bitmap h11.

[0075] The generation processing unit 150c repeatedly executes the above process for positions equal to or greater than position "29" in the bitmap b1, thereby generating a hashed bitmap h11.

[0076] An example of a process for generating a base "31" hashed bitmap h12 from a bitmap b1 will be described. First, the generation processing unit 150c copies the information at positions "0 to 30" of the bitmap b1 to the hashed bitmap h12. Next, since the remainder when the bit position "35" of the bitmap b1 is divided by the base "31" is "4," the position "35" of the bitmap b1 is associated with the position "4" of the hashed bitmap h12. Because a "1" is set at position "35" of the bitmap b1, the generation processing unit 150c sets a "1" at position "4" of the hashed bitmap h12.

[0077] The remainder when the bit position "42" of bitmap b1 is divided by the low "31" is "11", so position "42" of bitmap b1 corresponds to position "11" of hashed bitmap h12. Because "1" is set to position "42" of bitmap b1, the generation processing unit 150c sets "1" to position "11" of hashed bitmap h12.

[0078] The generation processing unit 150c repeatedly executes the above process for positions equal to or greater than position "31" in the bitmap b1, thereby generating a hashed bitmap h12.

[0079] The generation processor 150c hashes the word index 140c by compressing each row of the word index 140c using the folding technique described above. The hashed bitmaps of bases "29" and "31" are given information about the row (type of encoded word) of the original bitmap from which they were generated. While FIG. 10 illustrates the case where the generation processor 150c hashes the word index 140c, the synonym index 140d and the synonym sentence index 140e are also hashed in the same manner.

[0080] Returning to the description of FIG. 3, the extraction unit 150d calculates compressed vectors of multiple sentences included in the search query 20A. First, the extraction unit 150d obtains compressed vectors of multiple words included in one sentence from the dimension-compressed word vector table 140b, and restores the compressed vectors of the obtained words to 200-dimensional vectors. The compressed vectors in the dimension-compressed word vector table 140b are calculated using the values ​​of the basis vector a1e1, the basis vector a 67 e 67 The value of the basis vector a 133 e 133 is a vector whose values ​​are the dimension values.

[0081] 11 is a diagram for explaining dimension restoration. In FIG. 11, as an example, the basis vectors a1e1 and a2 are divided by the prime number "3". 67 e 67 , basis vector a 131 e131 Based on this, the basis vector a 45 e 45 The extraction unit 150d extracts the basis vector a 45 e 45 For basis vector a1e1, basis vector a 67 e 67 , basis vector a 131 e 131 By accumulating the orthogonal transformed values, the basis vector a 45 e 45 Restore the value of

[0082] The extraction unit 150d extracts the basis vector a 45 e 45 Similarly, by repeatedly executing the above process, the three-dimensional compressed vector is restored to a 200-dimensional vector.

[0083] Next, the extraction unit 150d calculates a sentence vector by integrating the vectors of multiple words included in one sentence using the dimension-reduced word table 140b. The extraction unit 150d similarly calculates sentence vectors for other sentences included in the search query 20A. The extraction unit 150d also calculates a "query vector" for the search query 20A by integrating the vectors of multiple sentences included in the search query 20A.

[0084] The extraction unit 150d sorts the values ​​of each dimension of the query vector in descending order and identifies the top "feature dimensions." Of the multiple sentences included in the search query 20A, the extraction unit 150d extracts, as characteristic sentences 22, sentences that contain many vector values ​​of the feature dimensions. Furthermore, of the multiple words included in the search query 20A, the extraction unit 150d extracts, as characteristic words 21, words that contain many vector values ​​of the feature dimensions. The extraction unit 150d outputs information on the characteristic words 21 and information on the characteristic sentences 22 to the identification unit 150e.

[0085] The identification unit 150e compares the compression vector of the characteristic word 21 with the compression vector of the synonym index 140d, and identifies the compression vector of the synonym index 140d whose similarity to the compression vector of the characteristic word 21 is equal to or greater than a threshold. The identification unit 150e searches the multiple text-compressed files 10B for a text-compressed file corresponding to the characteristic word 21, based on the offset corresponding to the identified compression vector, and generates the found text-compressed file as a first candidate list 31.

[0086] When the identification unit 150e calculates the similarity between the compressed vector of the characteristic word 21 and the compressed vector of the synonym index 140d, it uses formula (2). Here, the compressed vector of the synonym index 140d whose similarity to the compressed vector of the characteristic word 21 is equal to or greater than a threshold is referred to as a "similar compressed vector."

[0087] When there are multiple similar compressed vectors, the identification unit 150e sorts the similar compressed vectors in descending order of similarity and ranks the similar compressed vectors in descending order of similarity. When generating the first candidate list 31, the identification unit 150e registers the searched text compressed files in the first candidate list 31 based on the offsets corresponding to the similar compressed vectors with the greatest similarity. The identification unit 150e may also register the text compressed files in the first candidate list 31 in rank order.

[0088] The identification unit 150e compares the compressed vector of the characteristic sentence 22 with the compressed vector of the synonym sentence index 140e, and identifies the compressed vector of the synonym sentence index 140e whose similarity to the compressed vector of the characteristic sentence 22 is equal to or greater than a threshold. The identification unit 150e searches the multiple text compressed files 10B for a text compressed file corresponding to the characteristic sentence 22, based on the offset corresponding to the identified compressed vector, and generates the searched text compressed file as a second candidate list 32.

[0089] The identification unit 150e decrypts each compressed text file 10B registered in the first candidate list 31 based on the dictionary information 15 and the dynamic dictionary 140g, and outputs and displays the decrypted first candidate list 31 on the display unit 130. The identification unit 150e may also transmit the decrypted first candidate list 31 to the external device that transmitted the search query 20A.

[0090] When the identification unit 150e calculates the similarity between the compressed vector of the characteristic sentence 22 and the compressed vector of the synonymous sentence index 140e, it uses formula (2). Here, the compressed vector of the synonymous sentence index 140e whose similarity to the compressed vector of the characteristic sentence 22 is equal to or greater than a threshold is referred to as a "similar compressed vector."

[0091] When there are multiple similar compressed vectors, the identification unit 150e sorts the similar compressed vectors in descending order of similarity and ranks the similar compressed vectors in descending order of similarity. When generating the second candidate list 32, the identification unit 150e registers the searched text compressed files in the second candidate list 32 based on the offsets corresponding to the similar compressed vectors with the greatest similarity. The identification unit 150e may also register the text compressed files in the first candidate list 31 in rank order.

[0092] The identification unit 150e decrypts each compressed text file 10B registered in the second candidate list 32 based on the dictionary information 15 and the dynamic dictionary 140g, and outputs and displays the decrypted second candidate list 32 on the display unit 130. The identification unit 150e may also transmit the decrypted second candidate list 32 to the external device that transmitted the search query 20A.

[0093] Incidentally, when the synonym index 140d and the synonym sentence index 140e are hashed, the identification unit 150e restores the hashed bitmaps. Fig. 12 is a diagram for explaining the process of restoring the hashed bitmaps. Here, as an example, a case will be described in which the identification unit 150e restores the bitmap b1 based on the hashed bitmap h11 and the hashed bitmap h12.

[0094] The specification unit 150e generates an intermediate bitmap h11' from the hashed bitmap h11 of base 29. The specification unit 150e copies the values ​​at positions 0 to 28 of the hashed bitmap h11 to positions 0 to 28 of the intermediate bitmap h11', respectively.

[0095] For values ​​from position 29 onwards in the intermediate bitmap h11', the specification unit 150e repeatedly executes a process of copying the values ​​at positions 0 to 28 of the hashed bitmap h11 for each "29". In the example shown in Fig. 12, the values ​​at positions 0 to 14 of the hashed bitmap h11 are copied to positions 29 to 43 of the intermediate bitmap h11'.

[0096] The specification unit 150e generates an intermediate bitmap h12' from the hashed bitmap h12 with a base of 31. The specification unit 150e copies the values ​​at positions 0 to 30 of the hashed bitmap h12 to positions 0 to 30 of the intermediate bitmap h12', respectively.

[0097] For values ​​from position 31 onwards in the intermediate bitmap h12', the specification unit 150e repeatedly executes a process of copying the values ​​at positions 0 to 30 of the hashed bitmap h12 for each "31". In the example shown in Fig. 12, the values ​​at positions 0 to 12 of the hashed bitmap h12 are copied to positions 31 to 43 of the intermediate bitmap h12'.

[0098] After generating the intermediate bitmap h11' and the intermediate bitmap h12', the identification unit 150e performs an AND operation on the intermediate bitmap h11' and the intermediate bitmap h12' to restore the bitmap b1 before hashing. The identification unit 150e repeats the same process on the other hashed bitmaps to restore each bitmap corresponding to the code of the word (restores the synonym index 140d and the synonym sentence index 140e).

[0099] The graph generating unit 150f is a processing unit that generates graph information based on the specified text file 10A when it receives a specification of the text file 10A (or the compressed text file 10B) via the input unit 120 or the like. FIG. 13 is a diagram illustrating an example of graph information. Graph G10 shown in FIG. 13 is a graph illustrating the position corresponding to the compression vector of each word included in the text file 10A and the distribution state of the word. Graph G11 is a graph illustrating the position corresponding to the compression vector of each sentence included in the text file 10A and the transition state of the sentence. Graph G12 is a graph illustrating the position corresponding to the compression vector obtained by adding up multiple sentence vectors of the text file 10A. The horizontal axis of graphs G10 to G12 corresponds to the first dimension of the compression vector, and the vertical axis corresponds to the second dimension (a dimension different from the first dimension). For example, when graphing a university syllabus (lecture summary), the horizontal axis represents the era or year, and the vertical axis represents the dimension related to the region or location. It is assumed that the first and second dimensions are set in advance, and their values ​​are respectively accumulated and transformed from the three-dimensional compressed vector by orthogonal transformation.

[0100] An example of the process by which the graph generation unit 150f generates graph G10 will be described. The graph generation unit 150f performs lexical analysis on character strings included in the text file 10A and extracts words sequentially from the beginning. The graph generation unit 150f compares the extracted words with the dimension-compressed word vector table 140b to identify compressed vectors, and generates graph G10 by repeatedly executing a process of plotting points from the identified compressed vectors at positions on graph G10 corresponding to the first and second dimension values.

[0101] An example of the process by which the graph generation unit 150f generates the graph G11 will be described. The graph generation unit 150f performs lexical analysis on character strings included in the text file 10A and extracts sentences sequentially from the beginning. The graph generation unit 150f compares each word included in the sentence with the dimension-reduced word vector table 140b to identify the compressed vector of the word, and performs a process for calculating the compressed vector of the sentence for each sentence by integrating the words included in the sentence. The graph generation unit 150f generates the graph 10 by repeatedly performing a process of plotting points on the graph G11 corresponding to the first and second dimension values ​​for the compressed vector of each sentence. The graph generation unit 150f may connect the points on the graph G11 according to the order in which the sentences appear in the text file 10A.

[0102] An example of the process by which the graph generation unit 150f generates the graph G12 will be described. The graph generation unit 150f performs lexical analysis on character strings included in the text file 10A and extracts sentences sequentially from the beginning. The graph generation unit 150f compares each word included in the sentence with the dimension-compressed word vector table 140b to identify the compressed vector of the word, and executes a process for calculating the compressed vector of the sentence for each sentence by accumulating the words included in the sentence. The graph generation unit 150f also calculates the compressed vector of the text file 10A by accumulating the compressed vector of each sentence. The graph generation unit 150f generates the graph G12 by plotting the compressed vector of the text file 10A at positions on the graph G11 corresponding to the first and second dimension values.

[0103] In the above description, the graph generation unit 150f generates the graphs G10 to G12 separately, but the graph generation unit 150f may also generate the graphs G10 to G12 simultaneously. For example, the graph generation unit 150f may perform a lexical analysis on a character string included in the text file 10A, extract words in order from the beginning, and calculate the compression vectors of the sentence and the compression vectors of the text file 10A together in the process of identifying the compression vectors.

[0104] Next, an example of a processing procedure of the information processing device 100 according to this embodiment will be described. Fig. 14 is a flowchart (1) showing the processing procedure of the information processing device according to this embodiment. The receiving unit 150a of the information processing device 100 receives the text file 10A and registers it in the storage unit 140 (step S101).

[0105] The dimension reduction unit 150b of the information processing device 100 acquires the word vector table 140a (step S102). The dimension reduction unit 150b reduces the dimensions of each vector in the word vector table to generate the dimension-reduced word vector table 140b (step S103).

[0106] When compressing the text file 10A, the generation processing unit 150c of the information processing device 100 uses the dimension-compressed word vector table 140b to generate a word index 140c, a synonym index 140d, a synonym sentence index 140e, a sentence vector table 140f, and a dynamic dictionary 140g (step S104).

[0107] The generation processing unit 150c registers the word index 140c, the synonym index 140d, the synonym sentence index 140e, the sentence vector table 140f, and the dynamic dictionary 140g in the storage unit 140, and generates the compressed text file 10B (step S105).

[0108] 15 is a flowchart (2) showing the processing procedure of the information processing device according to this embodiment. The receiving unit 150a of the information processing device 100 receives the search query 20A (step S201). The extracting unit 150d of the information processing device 100 calculates a compressed vector of each sentence included in the search query 20A based on the dimension-reduced word vector table 140b (step S202).

[0109] The extraction unit 150d restores the dimension of the compressed vector of each sentence to 200 dimensions and identifies the feature dimension (step S203).The extraction unit 150d extracts feature words and feature sentences based on the feature dimension and identifies the compressed vector of the feature word and the compressed vector of the feature sentence (step S204).

[0110] The identification unit 150e of the information processing device 100 generates a first candidate list 31 based on the compressed vectors of the characteristic words and the synonym index, and outputs it to the display unit 130 (step S205). The identification unit 150e generates a second candidate list 32 based on the compressed vectors of the characteristic sentences and the synonym sentence index 140e, and outputs it to the display unit 130 (step S206).

[0111] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 generates a dimension-compressed word vector table 140b by dimensionally compressing the word vector table 140a, and when compressing the text file 10A, generates a synonym index 140d and a synonym sentence index 140e. The synonym index 140d is information that assigns the same compressed vector to each word belonging to the same synonym and defines the position where the word (synonym) corresponding to the compressed vector appears. Furthermore, the synonym sentence index 140e is information that assigns the same compressed vector to each sentence belonging to the same synonym sentence and defines the position where the sentence (synonym sentence) corresponding to the compressed vector appears. Therefore, the amount of data can be reduced compared to the conventional method of assigning a 200-dimensional vector to each word.

[0112] When a search query 20A is given, the information processing device 100 identifies the feature dimension of the search query 20A and identifies the feature word 21 and feature sentence 22 for which the vector value of the feature dimension is maximum. The information processing device 100 generates a first candidate list 31 based on the compressed vector of the feature word 21 and the synonym index 140d. The information processing device 100 generates a second candidate list 32 based on the compressed vector of the feature sentence 22 and the synonym sentence index 140e. Because the compressed vectors used in the feature word 21, feature sentence 22, synonym index 140d, and synonym sentence index 140e are three-dimensional vectors, it is possible to detect a compressed text file 10B that includes words and sentences similar to the search query 20A while suppressing the cost of similarity calculation.

[0113] The information processing device 100 generates and displays a graph G10 based on the compression vectors of multiple words included in the text file 10A, a graph G11 based on the compression vectors of multiple sentences, and a graph G12 based on the compression vector of the text file 10A. This makes it possible to visualize words, sentences, and text files (text).

[0114] Incidentally, the information processing device 100 according to the present embodiment uses one synonym index 140d to detect compressed text files 10B containing characteristic words extracted from the search query 20A and generates the first candidate list 31. However, the present invention is not limited to this. The information processing device 100 may generate multiple synonym indexes 140d with different granularities (different classification levels) and generate the first candidate list 31 using the multiple synonym indexes 140d.

[0115] Fig. 16 is a diagram showing an example of multiple synonym indices generated by the generation processing unit. Fig. 16 illustrates an example in which three synonym indices 140d-1, 140d-2, and 140d-3 are generated. A first reference value, a second reference value, and a third reference value are set for the synonym indices 140d-1, 140d-2, and 140d-3, respectively. The magnitude relationship between the reference values ​​is first reference value<second reference value<third reference value. The synonym index 140d-1 has the smallest granularity, followed by the synonym index 140d-2 and the synonym index 140d-3, in that order.

[0116] In the process of scanning and compressing words in the text file 10A from the beginning, the generation processing unit 150c repeatedly executes a process of acquiring, from the dimension-compressed word vector table 140b, compressed vectors corresponding to words to be compressed.

[0117] The generation processing unit 150c calculates the similarity of each compressed vector and determines that a group of compressed vectors whose similarity is equal to or greater than a first reference value is a synonym. The generation processing unit 150c identifies the average value of multiple compressed vectors included in the same group as a representative value of the multiple compressed vectors included in the same group, and sets a flag "1" in the synonym index 140d-1 based on the representative value (compressed vector) and the offset of the word corresponding to the compressed vector. The generation processing unit 150c sets each flag in the synonym index 140d-1 by repeatedly performing the above process for each group.

[0118] The generation processing unit 150c calculates the similarity of each compressed vector and determines that a group of compressed vectors whose similarity is equal to or greater than a second reference value is a synonym. The generation processing unit 150c identifies the average value of multiple compressed vectors included in the same group as a representative value of the multiple compressed vectors included in the same group, and sets a flag "1" in the synonym index 140d-2 based on the representative value (compressed vector) and the offset of the word corresponding to the compressed vector. The generation processing unit 150c sets each flag in the synonym index 140d-2 by repeatedly performing the above process for each group.

[0119] The generation processing unit 150c calculates the similarity of each compressed vector and determines that a group of compressed vectors whose similarity is equal to or greater than a third reference value is a synonym. The generation processing unit 150c identifies the average value of multiple compressed vectors included in the same group as a representative value of the multiple compressed vectors included in the same group, and sets a flag "1" in the synonym index 140d-3 based on the representative value (compressed vector) and the offset of the word corresponding to the compressed vector. The generation processing unit 150c sets each flag in the synonym index 140d-3 by repeatedly performing the above process for each group.

[0120] The identification unit 150e compares the compressed vectors of the characteristic words 21 extracted by the extraction unit 150d with the synonym indices 140d-1 to 140d-3, and identifies compressed vectors from the synonym indices 140d-1 to 140d-3 whose similarity to the compressed vectors of the characteristic words 21 is equal to or greater than a threshold.

[0121] The identification unit 150e searches for multiple text compressed files (first text compressed files) corresponding to the offset based on the offset of the compressed vector of the synonym index 140d-1, whose similarity to the compressed vector of the characteristic word 21 is equal to or greater than a threshold. The identification unit 150e searches for multiple text compressed files (second text compressed files) corresponding to the offset based on the offset of the compressed vector of the synonym index 140d-2, whose similarity to the compressed vector of the characteristic word 21 is equal to or greater than a threshold. The identification unit 150e searches for multiple text compressed files (third text compressed files) corresponding to the offset based on the offset of the compressed vector of the synonym index 140d-3, whose similarity to the compressed vector of the characteristic word 21 is equal to or greater than a threshold.

[0122] The identification unit 150e may register the first to third text compressed files in the first candidate list 31, or may register the text compressed file that is detected the largest number of times among the first to third text compressed files in the first candidate list 31.

[0123] Alternatively, the identification unit 150e may first use the synonym index 140d-3 with the largest granularity to search for text-compressed files, and if the number of searched text-compressed files is less than a predetermined number, switch to the synonym index 140d-2 with the next largest granularity to search for text-compressed files. Furthermore, the identification unit 150e may use the synonym index 140d-2 to search for text-compressed files, and if the number of searched text-compressed files is less than a predetermined number, switch to the synonym index 140d-1 with the next largest granularity to search for text-compressed files. In this way, by switching the synonym indexes, the number of search result candidates can be adjusted.

[0124] In the above example, a case has been described in which a first reference value, a second reference value, and a third reference value are set for the synonym index 140d to generate synonym indexes 140d-1 to 140d-3 with different granularities. However, this is not limiting. The generation processing unit 150c may set a first reference value, a second reference value, and a third reference value for the synonym sentence index 140e to generate synonym sentence indexes with different granularities. Furthermore, a user may operate the input unit 120 or the like to change the first reference value, the second reference value, and the third reference value as appropriate. When the generation unit 150c receives a change to the first reference value, the second reference value, and the third reference value, the generation unit 150c may dynamically recreate the synonym index 140d and the synonym sentence index 140e with different granularities.

[0125] The dimension reduction unit 150b according to the first embodiment calculates the values ​​of the basis vectors of the three prime numbers "1," "67," and "131" divided by the prime number "3" to obtain one compressed vector for one word. However, this is not limited to this. For example, when calculating a compressed vector, the dimension reduction unit 150b may set basis vectors of multiple prime numbers divided by multiple types of prime numbers and calculate multiple types of compressed vectors for one word. For example, the dimension reduction unit 150b may calculate the basis vectors of the three prime numbers "1," "67," and "131" divided by the prime number "3," the basis vectors of the five prime numbers "1," "41," "79," "127," and "163" divided by the prime number "5," and the basis vectors of the seven prime numbers "1," "29," "59," "83," "113," "139," and "173" divided by the prime number "7," and register multiple types of compressed vectors for one word in the dimension-compressed word vector table 140b. When the generation processing unit 150d and the extraction processing unit 150d use the dimension-reduced word vector table 140b, they may selectively use any of the compressed vectors to generate a transposed index or extract characteristic words and characteristic sentences.

[0126] Next, an example of a hardware configuration of a computer that realizes the same functions as the information processing device 100 shown in this embodiment will be described. Fig. 17 is a diagram showing an example of a hardware configuration of a computer that realizes the same functions as the information processing device according to this embodiment.

[0127] 17, computer 500 has a CPU 501 that executes various types of arithmetic processing, an input device 502 that accepts data input from a user, and a display 503. Computer 500 also has a reading device 504 that reads programs and the like from a storage medium, and an interface device 505 that exchanges data with external devices and the like via a wired or wireless network. Computer 500 also has a RAM 506 that temporarily stores various types of information, and a hard disk drive 507. Each of devices 501 to 507 is connected to a bus 508.

[0128] The hard disk drive 507 stores a receiving program 507a, a dimension reduction program 507b, a generation processing program 507c, an extraction program 507d, a specification program 507e, and a graph generation program 507f. The CPU 501 reads out the receiving program 507a, the dimension reduction program 507b, the generation processing program 507c, the extraction program 507d, the specification program 507e, and the graph generation program 507f and expands them in the RAM 506.

[0129] The receiving program 507a functions as a receiving process 506a. The dimension reduction program 507b functions as a dimension reduction process 506b. The generation processing program 507c functions as a generation processing process 506c. The extraction program 507d functions as an extraction process 506d. The identification program 507e functions as an identification process 506e. The graph generation program 507f functions as a graph generation process 506f.

[0130] The processing of the reception process 506a corresponds to the processing of the reception unit 150a. The processing of the dimensional reduction process 506b corresponds to the processing of the dimensional reduction unit 150b. The processing of the generation processing process 506c corresponds to the processing of the generation processing unit 550c. The processing of the extraction process 506d corresponds to the processing of the extraction unit 150d. The processing of the identification process 506e corresponds to the processing of the identification unit 150e. The processing of the graph generation process 506f corresponds to the processing of the graph generation unit 150f.

[0131] It should be noted that each of the programs 507a to 507f does not necessarily have to be stored in the hard disk drive 507 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD disk, magneto-optical disk, or IC card that is inserted into the computer 500. Then, the computer 500 may read and execute each of the programs 507a to 507f. [Explanation of symbols]

[0132] 10A Text File 10B Text Compressed File 15 Dictionary information 20A Search Query 31 First Candidate List 32 Second candidate list 100 Information processing device 110 Communications Department 120 Input section 130 Display section 140 Storage section 140a Word Vector Table 140b dimensionality compressed word vector table 140c Word Index 140d Synonym Index 140e Synonymous Sentence Index 140f Sentence Vector Table 140g Dynamic Dictionary 150 control section 150a Reception 150b Dimensionality reduction part 150c Generation processing unit 150d Extraction part 150e Specific part 150f Graph generation section

Claims

1. Based on a multi-dimensional vector corresponding to the character string, the vector is decomposed into components for each dimension to generate a plurality of basis vectors, which are equally distributed in a circular pattern; selecting first basis vectors from the plurality of basis vectors, the number of which is a prime number of three or more; and calculating, for each of the first basis vectors having the number of prime numbers, a value of each of the first basis vectors having the number of prime numbers by integrating values ​​obtained by orthogonally transforming the remaining base vectors, excluding the first basis vector, with respect to the first basis vector; The multiple dimensions included in the vector are compressed into the dimensions of the prime numbers, and the values ​​of the dimensions of the prime numbers are set to the values ​​of the first basis vectors of the number of prime numbers calculated by the calculation process. A dimensionality reduction method characterized in that the processing is executed by a computer.

2. an arrangement unit that, based on a multi-dimensional vector corresponding to a character string, decomposes the vector into components for each dimension to generate a plurality of basis vectors, and distributes the vector equally in a circular pattern; a calculation unit that selects first basis vectors whose number is a prime number of three or more from the plurality of basis vectors, and calculates values ​​of each of the first basis vectors whose number is a prime number by integrating values ​​obtained by orthogonally transforming each of the remaining base vectors, excluding the first basis vector, with respect to the first basis vector; a setting unit that compresses multiple dimensions included in the vector into dimensions of the prime number, and sets each value of the dimensions of the prime number to each value of the first basis vector of the number of prime numbers calculated by the calculating process; An information processing device having the above.

3. Based on a multi-dimensional vector corresponding to the character string, the vector is decomposed into components for each dimension to generate a plurality of basis vectors, which are equally distributed in a circular pattern; selecting first basis vectors from the plurality of basis vectors, the number of which is a prime number of three or more; and calculating, for each of the first basis vectors having the number of prime numbers, a value of each of the first basis vectors having the number of prime numbers by integrating values ​​obtained by orthogonally transforming the remaining base vectors, excluding the first basis vector, with respect to the first basis vector; The multiple dimensions included in the vector are compressed into the dimensions of the prime numbers, and the values ​​of the dimensions of the prime numbers are set to the values ​​of the first basis vectors of the number of prime numbers calculated by the calculation process. A program that causes a computer to perform a process.

Images