Information processing program, information processing method, and information processing device

Abstract

This information processing device divides text data into words. On the basis of a static dictionary for defining the relationship between a static word and a vector, the information processing device identifies an unknown word indicating a word that does not correspond to a static word in the static dictionary, from among a plurality of words included in the text data. On the basis of definition information for defining the relationship between a unique word and a vector generated on the basis of a condition that a similar vector is set for similar unique words, the information processing device assesses whether or not a unique word corresponding to the unknown word is included in the definition information. If the unique word corresponding to the unknown word is included in the definition information, the information processing device assigns a vector that corresponds to the unique word to the unknown word. If the unique word corresponding to the unknown word is not included in the definition information, the information processing device assigns a vector to the unknown word on the basis of words positioned before and after the unknown word.

Description

[Technical field]

[0001] The present invention relates to an information processing program and the like. [Background technology]

[0002] A huge amount of data is registered in a database (DB), and it is required to appropriately search for the data desired by the user from the DB.

[0003] An example of a conventional server that performs data searches will be described. When a server performs a data search, the server searches at various granularities such as documents, terms, and sentences. For the sake of convenience, however, a server that searches for sentences similar to a search query will be described here.

[0004] The server calculates the vector of each sentence included in the DB using a static dictionary or the like that defines the vector of a word, and generates an inverted index that indicates the relationship between the vector of the sentence and the position of the sentence in the DB. For example, the server calculates the vector of a sentence by accumulating the vectors of words included in the sentence. When the server receives a search query, the server calculates the vector of the search query in the same way as when calculating the vector of the sentence, and compares the vector of the search query with the inverted index to identify the position of a sentence similar to the search query.

[0005] Here, the static dictionary is a dictionary that defines the relationship between a given word and a vector corresponding to this word. For unknown words that indicate words not set in the static dictionary, conventional servers assign vectors to the unknown words using a technique such as CBOW (Continuous Bag-Of-Words). CBOW is a technique that calculates the vector of a target word based on the words that appear before and after the target word. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2017-194762 A [Patent Document 2] JP 2018-060463 A [Patent Document 3] Special Publication No. 2017-518570 [Patent Document 4] Patent Publication No. 2021-005117 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the above-mentioned conventional techniques have a problem in that the accuracy of vectors set in data such as documents and sentences is low.

[0008] For example, in conventional techniques, when assigning a vector to an unknown word that does not exist in a static dictionary, it is possible to use a technique such as CBOW, but such a technique may not be able to assign an appropriate vector to the unknown word.

[0009] In one aspect, the present invention has an object to provide an information processing program, an information processing method, and an information processing device that can improve the accuracy of vectors set in data. [Means for solving the problem]

[0010] In the first proposal, the computer is made to execute the following process. The computer divides the text data into words. The computer identifies unknown words indicating words that do not correspond to static words in the static dictionary, among a plurality of words included in the text data, based on a static dictionary that defines the relationship between static words and vectors. The computer determines whether or not a unique word corresponding to the unknown word is included in the definition information, based on a vector generated based on a condition that similar unique words are set to similar vectors, and on definition information that defines the relationship between the unique words. If a unique word corresponding to the unknown word is included in the definition information, the computer assigns a vector corresponding to the definition information of the unique word to the unknown word. If a unique word corresponding to the unknown word is not included in the definition information, the computer assigns a vector to the unknown word based on the words arranged before and after the unknown word. Effect of the Invention

[0011] The precision of the vector set in the data can be improved. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram for explaining the process of the information processing device according to the present embodiment. [Diagram 2] FIG. 2 is a functional block diagram illustrating a configuration of an information processing device according to this embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of a data file. [Figure 4] FIG. 4 is a diagram illustrating an example of a data structure of the static dictionary. [Diagram 5] FIG. 5 is a diagram illustrating an example of a data structure of the definition information. [Figure 6] FIG. 6 is a diagram illustrating an example of a data structure of the dynamic dictionary. [Figure 7] FIG. 7 illustrates an example of a data structure of an inverted index. [Figure 8] FIG. 8 is a flowchart (1) showing a processing procedure of the information processing device according to the present embodiment. [Figure 9] FIG. 9 is a flowchart (2) showing the processing procedure of the information processing device according to the present embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing apparatus of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an information processing program, an information processing method, and an information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the embodiments. EXAMPLES

[0014] 1 is a diagram for explaining the processing of the information processing device according to the present embodiment. For example, the information processing device uses a static dictionary 142, definition information 143, and a dynamic dictionary 144.

[0015] The static dictionary 142 is a dictionary that defines the relationship between a predetermined word, a static code, and a vector corresponding to the predetermined word. The predetermined words included in the static dictionary 142 are words that appear with a predetermined frequency or more in predetermined text data such as various papers and novels in Aozora Bunko, and are selected in advance.

[0016] Definition information 143 includes a plurality of proper nouns, and each proper noun is classified into groups of similar proper nouns. Proper nouns are words selected by experts in each field, without being limited to frequency of occurrence. In the example shown in FIG. 1, definition information 143 includes group G1, which is a collection of proper nouns related to illnesses, and group G2, which is a collection of proper nouns of products of company A. Although not described in FIG. 1, definition information 143 includes proper nouns belonging to other groups.

[0017] It is assumed that a vector is assigned in advance from text including each proper noun to each proper noun included in the definition information 143 using a technique called Poincare Embeddings. For Poincare Embedding, for example, a technique described in the non-patent document "Valentin Khrulkov1 et al. "Hyperbolic Image Embeddings", Cornell University, April 3, 2019" may be used.

[0018] In Poincaré embedding, words (proper nouns) are embedded in a space called Poincaré space, and vectors are assigned according to the embedding position. Also, a feature of Poincaré embedding is that the more similar the words are, the closer they are to each other in the Poincaré space. Therefore, proper nouns in the same group are embedded in close positions in the Poincaré space, and are assigned similar vectors.

[0019] For example, vectors vb1, vb2, and vb3 are assigned to proper nouns of diseases XYZdisease, XYXdisease, and XXXdisease belonging to group G1, and each of the vectors vb1 to vb3 is an approximate vector. Vectors vc1, vc2, and vc3 are assigned to proper nouns of products of company A belonging to group G2, such as ABCproduct, ABBproduct, and ACCproduct, and each of the vectors vc1 to vc3 is an approximate vector.

[0020] The dynamic dictionary 144 is a dictionary that associates a dynamic code corresponding to an unknown word not registered in the static dictionary 142 with a vector of the unknown word, and each piece of information is registered by a process described later. Also, the dynamic dictionary 144 registers vectors corresponding to unknown words.

[0021] The information processing device of this embodiment executes the following process using the static dictionary 142, definition information 143, and dynamic dictionary 144 to calculate the vector of sentence 10. For example, sentence 10 is "According to the researcher, this XYZdisease can be caused by stress." The information processing device executes morphological analysis on sentence 10 to divide sentence 10 into a plurality of words. In the following explanation, explanations of commas, periods, and spaces will be omitted.

[0022] The information processing device compares the words included in sentence 10 with static dictionary 142, and for the words included in static dictionary 142, assigns vectors defined in static dictionary 142. For example, of the words included in sentence 10, "According," "to," "the," "researcher," "this," "can," "be," "caused," "by," and "stress" are words defined in static dictionary 142, and vectors va1 to va10 are assigned to them, respectively.

[0023] The information processing device compares the words included in sentence 10 with static dictionary 142, and identifies the words not included in static dictionary 142 as "unknown words." For example, if the word "XYZdisease" included in sentence 10 is not registered in static dictionary 142, "XYZdisease" becomes an unknown word. The information processing device compares the unknown word with definition information 143, and identifies whether or not a proper noun corresponding to the unknown word is included in definition information 143.

[0024] When a proper noun corresponding to an unknown word is included in definition information 143, the information processing device assigns a vector defined in definition information 143 to the unknown word. In the example shown in Fig. 1, a proper noun corresponding to the unknown word "XYZdisease" is included in definition information 143, so the information processing device assigns vector vb1 of the proper noun "XYZdisease" to the unknown word.

[0025] On the other hand, if the definition information 143 does not include a proper noun corresponding to the unknown word, the information processing device calculates a vector of the unknown word using a technique such as CBOW (Continuous Bag-Of-Words).

[0026] The information processing device registers the unknown words in the dynamic dictionary 144, assigns a dynamic code to the unknown words, associates the dynamic code with the vector assigned to the unknown words, and registers the dynamic code in the dynamic dictionary 144. The information processing device assigns the dynamic codes to the unknown words in order based on a plurality of dynamic codes prepared in advance.

[0027] The information processing device assigns a vector to each word through the above process, and multiplies each vector to calculate vector Vec1 of sentence 10. The information processing device compresses the words included in sentence 10 into static codes and dynamic codes based on static dictionary 141 and dynamic dictionary 144. Furthermore, when an unknown word in sentence 10 has already been registered in dynamic dictionary 144, the information processing device uses a vector corresponding to the unknown word (proper noun) registered in dynamic dictionary 144.

[0028] As described above, when the information processing device according to this embodiment assigns a vector to each word in sentence 10, the information processing device assigns a vector defined in static dictionary 141 to each word registered in static dictionary 141. For an unknown word included in sentence 10, if the unknown word corresponds to a proper noun included in definition information 143, the information processing device assigns a vector defined in definition information 143 to the unknown word, and if the unknown word does not correspond to a proper noun in definition information 143, the information processing device assigns a vector using CBOW.

[0029] For example, when a vector of an unknown word is calculated using CBOW, the value of the vector may vary depending on the words that appear before and after the unknown word. On the other hand, for proper nouns defined in the definition information 143, the vector value can be stabilized by calculating the vector in advance using Poincaré embedding. This can improve the accuracy of the vector of the sentence.

[0030] Next, an example of the configuration of the information processing device according to the present embodiment will be described. Fig. 2 is a functional block diagram showing the configuration of the information processing device according to the present embodiment. As shown in Fig. 2, the information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0031] The communication unit 110 is connected to an external device or the like by wire or wirelessly, and transmits and receives information to and from the external device or the like. For example, the communication unit 110 is realized by a NIC (Network Interface Card) or the like. The communication unit 110 may be connected to a network (not shown).

[0032] The input unit 120 is an input device that inputs various types of information to the information processing device 100. The input unit 120 corresponds to a keyboard, a mouse, a touch panel, etc. For example, a user may operate the input unit 120 to input a search query, etc.

[0033] The display unit 130 is a display device that displays information output from the control unit 150. The display unit 130 corresponds to a liquid crystal display, an organic EL (Electro Luminescence) display, a touch panel, etc. For example, search results corresponding to a search query are displayed on the display unit 130.

[0034] The storage unit 140 has a data file 141a, a compressed file 141b, a static dictionary 142, definition information 143, a dynamic dictionary 144, and an inverted index 145. The storage unit 140 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0035] The data file 141a corresponds to text data. Fig. 3 is a diagram showing an example of a data file. Fig. 3 shows text data in English, but is not limited to this, and may be text data in other languages, or may be a base sequence showing a genome or a chemical structural formula showing an organic compound.

[0036] The compressed file 141b is data obtained by compressing the data file 141a on a word-by-word basis based on the static dictionary 142 and the dynamic dictionary 144. The compressed file 141b is not shown in the figure.

[0037] The static dictionary 142 is a dictionary that defines the relationship between a predetermined word, a static code, and a vector corresponding to the predetermined word. Fig. 4 is a diagram showing an example of a data structure of the static dictionary. As shown in Fig. 4, the static dictionary 142 associates words, static codes, and vectors.

[0038] Vectors corresponding to a plurality of proper nouns are set in the definition information 143. Fig. 5 is a diagram showing an example of a data structure of the definition information. As shown in Fig. 5, the definition information 143 associates a group ID (Identification), a proper noun, and a vector.

[0039] The group ID is information that uniquely identifies the group to which the proper noun belongs. Proper nouns that belong to the same group are similar to each other. Proper nouns are words that are set by experts in each field, without being limited by frequency of occurrence, etc. The vector is a vector that is set for a proper noun, and is assigned in advance, for example, using a technique called Poincaré embedding.

[0040] The dynamic dictionary 144 is a dictionary that associates a dynamic code corresponding to an unknown word not registered in the static dictionary 142 with a vector of the unknown word. Fig. 6 is a diagram showing an example of a data structure of the dynamic dictionary. As shown in Fig. 6, the dynamic dictionary 144 has a dictionary table 144a and a dictionary table 144b.

[0041] In dictionary table 144a, proper nouns defined in definition information 143 among unknown words are associated with dynamic codes and vectors. The range of dynamic codes assigned to proper nouns is preset, for example, "E0000000h to EFFFFFFFh." "h" is a symbol indicating a hexadecimal number. Proper nouns are proper nouns defined in definition information 143. Vectors are vectors defined in definition information 143. Note that dictionary table 144a may be configured to store proper nouns in a storage area different from dictionary table 144a, and to indicate the position of the proper noun with a pointer.

[0042] In the dictionary table 144b, unknown words not defined in the definition information 143, such as new words and buzzwords, are associated with dynamic codes and vectors. The range of dynamic codes assigned to proper nouns is preset, for example, "F0000000h to FFFFFFFFh." "h" is a symbol indicating a hexadecimal number. Unknown words such as new words and buzzwords are unknown words not defined in the definition information 143. The vectors are vectors calculated by CBOW. Note that the dictionary table 144b may be configured so that unknown words such as new words and buzzwords are stored in a storage area different from the dictionary table 144b, and the position of the unknown word is indicated by a pointer.

[0043] The transposed index 145 is information that defines the relationship between the vector of a sentence and the offset of the code of the first word of the sentence. The code corresponds to a static code in the static dictionary 142 or a dynamic code in the dynamic dictionary 144. For example, offset "0" corresponds to the position of the first code in the compressed file 141b. Offset "N" corresponds to the position of the "N+1"th code from the beginning of the compressed file 141b.

[0044] FIG. 7 is a diagram showing an example of the data structure of a transposed index. In the transposed index 145 shown in FIG. 7, the horizontal axis indicates the offset of the compressed file 141b. The vertical axis corresponds to the sentence vector. For example, a flag "1" is set at the intersection of the row of the sentence vector "Vsen1" and the column of the offset "3". This indicates that the code of the first word of the sentence corresponding to the sentence vector "Vsen1" is located at the offset "3" of the compressed file 141b. It is assumed that the initial values ​​of the transposed index 145 are all set to 0.

[0045] Returning to the description of Fig. 2, the control unit 150 includes a generating unit 151 and a searching unit 152. The control unit 150 is realized by, for example, a central processing unit (CPU) or a micro processing unit (MPU). The control unit 150 may also be executed by, for example, an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0046] The generating unit 151 calculates vectors of sentences included in the data file 141a by executing the process described in Fig. 1. In addition, the generating unit 151 compresses the data file 141a into a compressed file 141b using the static dictionary 142 and the dynamic dictionary 144, and generates an inverted index 145.

[0047] The generating unit 151 acquires one sentence from the data file 141a based on a period, a punctuation mark, etc. The generating unit 151 executes the following process for the acquired sentence. Here, the process of the generating unit 151 will be explained using the sentence 10 explained in FIG. 1.

[0048] The generation unit 151 divides the sentence 10 into a plurality of words by performing a morphological analysis on the sentence 10. The generation unit 151 compares the words included in the sentence 10 with the static dictionary 142, and assigns vectors defined in the static dictionary 142 to the words included in the static dictionary 142. In the example described in FIG. 1, vectors va1 to va10 are assigned to "According", "to", "the", "researcher", "this", "can", "be", "caused", "by", and "stress", respectively.

[0049] The generation unit 151 compares the words included in the sentence 10 with the static dictionary 142, and identifies the words not included in the static dictionary 142 as "unknown words." For example, if the word "XYZdisease" included in the sentence 10 is not registered in the static dictionary 142, "XYZdisease" becomes an unknown word. If the unknown word "XYZdisease" is registered in the dynamic dictionary 144, the generation unit 151 assigns it to a vector corresponding to the unknown word.

[0050] On the other hand, if the unknown word “XYZdisease” is not registered in the dynamic dictionary 144, the generation unit 151 compares the unknown word with the definition information 143 to determine whether or not a proper noun corresponding to the unknown word is included in the definition information 143.

[0051] When a proper noun corresponding to an unknown word is included in definition information 143, generation unit 151 assigns a vector defined in definition information 143 to the unknown word. Furthermore, generation unit 151 assigns an unassigned dynamic code from "E0000000h to EFFFFFFFh" to the unknown word, associates the dynamic code, the unknown word (proper noun), and the vector, and registers them in the dynamic dictionary.

[0052] On the other hand, when a proper noun corresponding to an unknown word such as a new word or a buzzword is not included in the definition information 143, the generation unit 151 calculates a vector of the unknown word using a technique such as CBOW. In addition, the generation unit 151 assigns an unassigned dynamic code from "F0000000h to FFFFFFFFh" to the unknown word, associates the dynamic code, the unknown word (such as a new word or a buzzword), and a vector (vector by CBOW) with each other, and registers them in the dynamic dictionary 144.

[0053] The generation unit 151 assigns a vector to each word and unknown word of the sentence 10 through the above process, and calculates the vector Vsen1 of the sentence 10 by integrating each assigned vector. The generation unit 151 compresses the words of the sentence 10 by converting them into static codes and dynamic codes based on the static dictionary 142 and dynamic dictionary 144, and registers the compressed sentence 10 in the compression file 141b. The generation unit 151 sets the relationship between the offset of the leading code of the compressed sentence 10 and the vector of the sentence 10 in the transposed index 145.

[0054] The generating unit 151 repeatedly executes the above process for each sentence included in the data file 141a to generate the compressed file 141b and the transposed index 145. When the generating unit 151 registers the compressed sentence in the compressed file 141b, the generating unit 151 may place a code indicating a delimiter between the last code of the previous compressed sentence and the first code of the current compressed sentence. If the last code of the compressed sentence is a unique code such as a period, it is not necessary to place a code indicating a delimiter.

[0055] Returning to the description of Fig. 2, the search unit 152 is a processing unit that searches for a sentence similar to the search query when the search unit 152 receives the search query. An example of the process of the search unit 152 will be described below. In this embodiment, the search query will be described as a sentence having a plurality of words.

[0056] The search unit 152 executes a process of calculating a vector of a search query and a process of searching for a sentence similar to the search query by using the vector of the search query.

[0057] The process of calculating a vector of a search query by the search unit 152 will be described. Upon receiving a search query, the search unit 152 executes morphological analysis to divide the search query into a plurality of words. The search unit 152 compares the divided words with the static dictionary 142, and assigns a vector defined in the static dictionary 142 to the words included in the static dictionary 142. If there is an unknown word not included in the static dictionary 142 among the divided words, the search unit 152 compares the unknown word with the dynamic dictionary 144, and assigns a vector set in the dynamic dictionary 144 to the unknown word included in the dynamic dictionary 144.

[0058] When the corresponding unknown word is not set in the dynamic dictionary 144, the search unit 152 assigns a vector corresponding to the unknown word using the definition information 143 in the same manner as the generation unit 151.

[0059] For example, when a proper noun corresponding to an unknown word is included in definition information 143, search unit 152 assigns a vector defined in definition information 143 to the unknown word. In addition, search unit 152 assigns an unassigned dynamic code from "E0000000h to EFFFFFFFh" to the unknown word, and registers the dynamic code, the unknown word (proper noun), and the vector in the dynamic dictionary in association with each other.

[0060] When a proper noun corresponding to an unknown word such as a new word or a buzzword is not included in the definition information 143, the search unit 152 calculates a vector of the unknown word using a technique such as CBOW. In addition, the search unit 152 assigns an unassigned dynamic code from "F0000000h to FFFFFFFFh" to the unknown word, associates the dynamic code with the unknown word (such as a new word or a buzzword), and registers the vector (vector by CBOW) in the dynamic dictionary 144.

[0061] Through the above process, the search unit 152 assigns a vector to each word and unknown word in the search query, and calculates the vector of the search query by integrating the assigned vectors. In the following description, the vector of the search query is referred to as a "search vector."

[0062] Next, a process in which the search unit 152 searches for sentences similar to a search query will be described. The search unit 152 identifies a sentence vector similar to the search vector based on the search vector and the vector of each sentence in the transposed index. For example, the search unit 152 calculates the cosine similarity between the search vector and the sentence vector, and identifies a sentence vector whose cosine similarity is equal to or greater than a threshold. In the following description, a sentence vector whose cosine similarity with the search vector is equal to or greater than a threshold will be referred to as a "specific vector."

[0063] The search unit 152 scans the row of a specific vector among the vectors of each sentence in the transposed index 145 to identify the offset of the sentence of the specific vector. In the following description, the offset of the sentence of the specific vector is referred to as a "specific offset." The search unit 152 extracts the compressed sentence from the position of the compressed file 141b corresponding to the specific offset, and decodes the compressed sentence based on the static dictionary 142 and the dynamic dictionary 144. The search unit 152 outputs the decoded sentence to the display unit 130 for display. When the information processing device 100 (search unit 152) receives a search query from an external device via a network, it transmits the decoded sentence to the external device.

[0064] Next, an example of a processing procedure of the information processing device according to the present embodiment will be described. Fig. 8 is a flowchart (1) showing the processing procedure of the information processing device according to the present embodiment. As shown in Fig. 8, the generation unit 151 of the information processing device 100 acquires a sentence from the data file 141a (step S101).

[0065] The generating unit 151 performs a morphological analysis on the acquired sentence (step S102). The generating unit 151 assigns vectors to words based on the static dictionary 142 (step S103).

[0066] The generation unit 151 determines whether the unknown word is registered in the dynamic dictionary 144 (step S104). If the unknown word is registered in the dynamic dictionary 144 (step S104, Yes), the generation unit 151 assigns a vector to the unknown word based on the dynamic dictionary 144 (step S105), and proceeds to step S110.

[0067] On the other hand, if the unknown word is not registered in the dynamic dictionary 144 (step S104, No), the generation unit 151 determines whether or not the unknown word is registered in the definition information 143 (step S106). If the unknown word is not registered in the definition information 143 (step S106, No), the generation unit 151 calculates a vector of the unknown word based on the CBOW (step S107), and proceeds to step S109.

[0068] On the other hand, if the unknown word is registered in the definition information 143 (step S106, Yes), the generation unit 151 acquires a vector of the unknown word based on the definition information 143 (step S108). The generation unit 151 associates the unknown word, the dynamic code, and the vector, and registers them in the dynamic dictionary 144 (step S109).

[0069] The generating unit 151 calculates a vector of the sentence by accumulating the vectors assigned to each word included in the sentence (step S110). The generating unit 151 compresses the sentence based on the static dictionary 142 and the dynamic dictionary 144, and registers the compressed file 141b (step S111).

[0070] The generating unit 151 associates the offset of the compressed sentence with the vector of the sentence and updates the transposed index 145 (step S112). If the generating unit 151 has not selected all the sentences in the data file 141a (step S113, No), the generating unit 151 proceeds to step S101. On the other hand, if the generating unit 151 has selected all the sentences in the data file 141a (step S113, Yes), the generating unit 151 ends the process.

[0071] 9 is a flowchart (2) showing the processing procedure of the information processing device according to the present embodiment. As shown in FIG 9, the search unit 152 of the information processing device 100 receives a search query (step S201).

[0072] The search unit 152 performs a morphological analysis on the search query (step S202). The search unit 152 assigns vectors to words based on the static dictionary 142 (step S203).

[0073] The search unit 152 determines whether the unknown word is registered in the dynamic dictionary 144 (step S204). If the unknown word is registered in the dynamic dictionary 144 (step S204, Yes), the search unit 152 assigns a vector to the unknown word based on the dynamic dictionary 144 (step S205), and proceeds to step S210.

[0074] On the other hand, if the unknown word is not registered in the dynamic dictionary 144 (step S204, No), the search unit 152 determines whether or not the unknown word is registered in the definition information 143 (step S206). If the unknown word is not registered in the definition information 143 (step S206, No), the search unit 152 calculates a vector of the unknown word based on the CBOW (step S207), and proceeds to step S209.

[0075] On the other hand, if the unknown word is registered in the definition information 143 (step S206, Yes), the search unit 152 acquires a vector of the unknown word based on the definition information 143 (step S208). The search unit 152 associates the unknown word, the dynamic code, and the vector, and registers them in the dynamic dictionary 144 (step S209).

[0076] The search unit 152 calculates a search vector by accumulating vectors assigned to each word included in the search query (step S210). The search unit 152 identifies a specific vector and a specific offset based on the cosine similarity between the search vector and the vector of each sentence in the transposed index 145 (step S211).

[0077] The search unit 152 obtains the compressed sentence corresponding to the specific offset from the compressed file 141b (step S212). The search unit 152 decodes the compressed sentence based on the static dictionary 142 and the dynamic dictionary 144 (step S213). The search unit 152 outputs the decoded sentence (step S214).

[0078] Next, the effects of the information processing device 100 according to this embodiment will be described. When the information processing device 100 assigns a vector to each word in a sentence, the information processing device 100 assigns a vector defined in the static dictionary 141 to each word registered in the static dictionary 141. For an unknown word included in a sentence, if the unknown word corresponds to a proper noun included in the definition information 143, the information processing device 100 assigns a vector defined in the definition information 143 to the unknown word, and if the unknown word does not correspond to a proper noun in the definition information 143, the information processing device 100 assigns a vector using CBOW.

[0079] For example, when a vector of an unknown word is calculated using CBOW, the value of the vector may vary depending on the words that appear before and after the unknown word. On the other hand, for proper nouns defined in the definition information 143, the vector value can be stabilized by calculating the vector in advance using Poincaré embedding. This can improve the accuracy of the vector of the sentence.

[0080] The information processing device 100 registers the relationship between the unknown word and the vector assigned to the unknown word in the dynamic dictionary 144. This makes it possible to assign a vector to the same unknown word using the dynamic dictionary 144.

[0081] The information processing device 100 executes a process of assigning vectors to sentences included in the data file 141a, compresses the sentences, and registers them in the compressed file 141b. The information processing device 100 also associates the vectors of the compressed sentences with the offsets of the sentences, and registers them in the transposed index 145. By using the transposed index 145, it is possible to easily search for sentences.

[0082] When the information processing device 100 receives a search query, it divides the search data into words, and calculates a vector of the search query based on the divided words, the static dictionary 142, the definition information 143, and the dynamic dictionary 144. Furthermore, the information processing device 100 identifies the position of a sentence similar to the search query based on the vector of the search data and each vector of the transposed index 145. This makes it possible to search for sentences similar to the search query with high accuracy.

[0083] Incidentally, in the above embodiment, the case where a vector is calculated at the granularity of a sentence has been described, but it is also possible to calculate a vector at the granularity of a document. For example, a vector of a document can be calculated by accumulating vectors of sentences contained in the document. In addition, by further setting a base sequence indicating a genome, a rational formula of a chemical substance, an error code of a program, etc. as a proper noun in the definition information 143, it is also possible to improve the accuracy of vectors of sentences including genomes, rational formulas of chemical substances, and error codes of programs in a similar manner.

[0084] Next, an example of a hardware configuration of a computer that realizes the same functions as the information processing device 100 described in the above embodiment will be described. Fig. 10 is a diagram showing an example of a hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0085] 10, the computer 200 has a CPU 201 that executes various arithmetic processes, an input device 202 that accepts data input from a user, and a display 203. The computer 200 also has a communication device 204 that transmits and receives data to and from external devices, etc., via a wired or wireless network, and an interface device 205. The computer 200 also has a RAM 206 that temporarily stores various information, and a hard disk device 207. The devices 201 to 207 are connected to a bus 208.

[0086] The hard disk device 207 includes a generation program 207a and a search program 207b. The CPU 201 reads out the programs 207a and 207b and develops them in the RAM 206.

[0087] The generation program 207a functions as the generation process 206a, and the search program 207b functions as the search process 206b.

[0088] The processing of the generation process 206a corresponds to the processing of the generation unit 151. The processing of the search process 206b corresponds to the processing of the search unit 152.

[0089] It should be noted that each of the programs 207a and 207b does not necessarily have to be stored in the hard disk device 207 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a and 207b. [Explanation of symbols]

[0090] 100 Information processing device 110 Communications Department 120 Input section 130 Display section 140 Storage section 141a Data File 141b Compressed File 142 Static Dictionary 143 Definition information 144 Dynamic Dictionary 145 Inverted Index 150 Control section 151 Generation part 152 Search Department

Claims

1. Split the text data into words, assigning a static code and a vector in the static dictionary to a word that matches a word in the static dictionary among a plurality of words included in the text data based on a static dictionary that defines a relationship between a static code, a word, and a vector; determining whether or not the unknown word matches the proper noun based on an unknown word indicating a word that does not match a word in the static dictionary among a plurality of words included in the text data and a definition table that defines a relationship between proper nouns and vectors; if the unknown word and the proper noun match, assigning a vector of the proper noun to the unknown word, setting a relationship between the unknown word and the assigned vector in a dynamic dictionary, and assigning a dynamic code to the unknown word; if the unknown word does not match the proper noun, dynamically assigning a vector to the unknown word, setting a relationship between the unknown word and the dynamically assigned vector in the dynamic dictionary, and assigning a dynamic code to the unknown word; calculating a partial vector by integrating vectors assigned to the words and the unknown words included in the text data; compressing the text data based on the static codes assigned to the words included in the text data and the dynamic codes assigned to the unknown words; An inverted index is generated that associates the partial vector, the compressed text data, and an offset of the text data. An information processing program that causes a computer to execute a process.

2. When search data is received, a vector of the search data is generated; Obtaining an offset of a partial vector similar to the vector of the search data based on the vector of the search data and the inverted index; Obtaining compressed text data from a position corresponding to the obtained offset; Decoding the acquired compressed text data based on the static dictionary and the dynamic dictionary; Output the decrypted result 2. The information processing program according to claim 1, further comprising: a program for causing a computer to execute a process.

3. Split the text data into words, assigning a static code and a vector of a word in the static dictionary to a word that matches a word in the static dictionary among a plurality of words included in the text data based on a static dictionary that defines a relationship between a static code, a word, and a vector; determining whether or not the unknown word matches the proper noun based on an unknown word indicating a word that does not match a word in the static dictionary among a plurality of words included in the text data and a definition table that defines a relationship between proper nouns and vectors; if the unknown word and the proper noun match, assigning a vector of the proper noun to the unknown word, setting a relationship between the unknown word and the assigned vector in a dynamic dictionary, and assigning a dynamic code to the unknown word; if the unknown word does not match the proper noun, dynamically assigning a vector to the unknown word, setting a relationship between the unknown word and the dynamically assigned vector in the dynamic dictionary, and assigning a dynamic code to the unknown word; calculating a partial vector by integrating vectors assigned to the words and the unknown words included in the text data; compressing the text data based on the static codes assigned to the words included in the text data and the dynamic codes assigned to the unknown words; An inverted index is generated that associates the partial vector, the compressed text data, and an offset of the text data. An information processing method characterized in that the processing is executed by a computer.

4. Split the text data into words, assigning a static code and a vector of a word in the static dictionary to a word that matches a word in the static dictionary among a plurality of words included in the text data based on a static dictionary that defines a relationship between a static code, a word, and a vector; determining whether or not the unknown word matches the proper noun based on an unknown word indicating a word that does not match a word in the static dictionary among a plurality of words included in the text data and a definition table that defines a relationship between proper nouns and vectors; if the unknown word and the proper noun match, assigning a vector of the proper noun to the unknown word, setting a relationship between the unknown word and the assigned vector in a dynamic dictionary, and assigning a dynamic code to the unknown word; if the unknown word does not match the proper noun, dynamically assigning a vector to the unknown word, setting a relationship between the unknown word and the dynamically assigned vector in the dynamic dictionary, and assigning a dynamic code to the unknown word; calculating a partial vector by integrating vectors assigned to the words and the unknown words included in the text data; compressing the text data based on the static codes assigned to the words included in the text data and the dynamic codes assigned to the unknown words; An inverted index is generated that associates the partial vector, the compressed text data, and an offset of the text data. An information processing device comprising a control unit for executing processing.

Images