Synonym extraction apparatus, synonym extraction method, and synonym extraction program

The synonym extraction device addresses context-dependent polysemy by using a multi-layered language model to calculate word and context similarities, improving the accuracy and stability of synonym identification.

JP2024165355APending Publication Date: 2024-11-28HITACHI LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023081494
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-17
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing synonym extraction methods fail to accurately account for context-dependent polysemy, leading to the extraction of non-synonymous words as synonyms due to excessive context influence, as seen in techniques like BERT-based lexical substitution.

Method used

A synonym extraction device that utilizes a processor and storage to learn context-dependent word representations by employing a language model with multiple layers, calculating word and context similarities to optimize synonym extraction, integrating these measures to determine the most likely synonyms.

Benefits of technology

This approach optimizes synonym extraction by balancing context independence and dependence, enhancing accuracy and stability through the use of multiple language models and layers, reducing the influence of context on synonym identification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024165355000001_ABST
    Figure 2024165355000001_ABST
Patent Text Reader

Abstract

To properly extract synonyms in consideration of context.SOLUTION: A synonym extraction apparatus is configured to access a language model trained to output synonyms of a word in a sentence, the language model including a first layer which converts a word constituting a sentence into a first embedded representation that is not dependent on context of the sentence and a second layer which generates a second embedded representation that is dependent on context of the sentence on the basis of a position in which the word appears, a sentence in which the word appears, and the first embedded representation of the word, the synonym extraction apparatus being configured to: calculate word similarity by generating a first embedded representation regarding a target word using the first layer and a first embedded representation regarding a synonym candidate, the synonym candidate being a word other than the target word in a target sentence; calculate context similarity by generating a second embedded representation regarding the target sentence using the first and second layers, and a second embedded representation regarding a synonym sentence, the synonym sentence being obtained by replacing the target word in the target sentence by the synonym candidate; and extract a synonym on the basis of the word similarity and the context similarity.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a synonym extraction device, a synonym extraction method, and a synonym extraction program for extracting synonyms. [Background technology]

[0002] Many question-answering systems, such as chatbots, compare the question from the user with expected questions preregistered in the system, and output the answer that best corresponds to the expected question. For example, in a question-answering system for administrative procedures, the text "I've moved, what should I do?" is registered as an expected question.

[0003] However, since users ask questions with various expressions even if they have the same intention, it is necessary to register as expected questions "I moved, what should I do?" and "I changed my address, what should I do?" in addition to "I moved, what should I do?". In this example, the variation of expected questions will be increased by using "relocating" and "change of address," which are synonyms of "moving." To improve the answering accuracy of a question answering system, it is desirable to prepare as many synonyms as possible for the words in the expected questions.

[0004] Non-Patent Document 1 discloses an efficient estimation of word representations in a vector space in which embedded representations of words are learned from a large amount of text and words with high similarity between the embedded representations are treated as synonyms. The embedded representations are real vectors of several hundred dimensions, and cosine is used to measure the similarity between embedded representations. Learning proceeds so that a target word can be predicted from surrounding words, or conversely, surrounding words can be predicted from a target word.

[0005] However, since surrounding words are ignored after learning, it was not possible to handle the ambiguity of words depending on the context. For example, the embedding of "moving" is the same in the context of "I moved, what should I do?" and in the context of "Please tell me the cost of moving." Therefore, in the former context, "moving" and "change of address" are synonymous, but in the latter context, they are not synonymous, and it was not possible to handle the ambiguity of words depending on the context.

[0006] Non-Patent Document 2 discloses BERT-based lexical replacement, which uses a language model trained from a large amount of text to obtain embedded expressions of words that depend on the context. The embedded expressions for the same word "moving" are different in "I moved, what should I do?" and "Please tell me the cost of moving." Therefore, the technology in Non-Patent Document 2 extracts different synonyms depending on the context, even for the same word. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] “Efficient Estimation of Word Representations in Vector Space”,In Proceedings of Workshop at ICLR,2013. [Non-Patent Document 2] “BERT-based Lexical Substitution”,ACL 2019,2019. Summary of the Invention [Problem to be solved by the invention]

[0008] However, in the technology of Non-Patent Document 2, the word embedding representation is too strongly influenced by the context, and even words that are not synonymous are often extracted as synonyms. For example, in the context of "Please tell me the cost of moving," "purchase" and "inquiry" are also extracted as synonyms for "moving."

[0009] An object of the present invention is to optimize synonym extraction in consideration of context. [Means for solving the problem]

[0010] A synonym extraction device according to one aspect of the invention disclosed in the present application includes a processor that executes a program and a storage device that stores the program, and is capable of accessing a language model trained to output synonyms of words in a sentence, the language model having a first layer and a second layer, the first layer converting words constituting the sentence into a first embedded representation that is independent of the context of the sentence, the second layer generating a second embedded representation that is dependent on the context of the sentence based on the appearance position of the word, the sentence in which the word appears, and the first embedded representation of the word, the processor treating words other than a target word in a target sentence as synonym candidates, and generating a first embedded representation for the target word and a first embedded representation for the synonym candidate using the first layer. a word similarity calculation process for generating a first embedded expression for the target sentence and a second embedded expression for the synonym sentence, and calculating a word similarity, which is the similarity between the target sentence and the synonym candidate, based on a first embedded expression for the target word and a first embedded expression for the synonym candidate; a context similarity calculation process for generating a second embedded expression for the target sentence and a second embedded expression for the synonym sentence using the first layer and the second layer, and calculating a context similarity, which is the similarity between the target sentence and the synonym sentence, based on the second embedded expression for the target sentence and the second embedded expression for the synonym sentence; and a synonym extraction process for extracting the synonym based on the word similarity and the context similarity. Effect of the Invention

[0011] According to the representative embodiment of the present invention, it is possible to optimize synonym extraction in consideration of the context. Problems, configurations, and effects other than those described above will become apparent from the following description of the examples. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a synonym extraction device according to this embodiment. [Diagram 2] Figure 2 is a schematic diagram of a language model called a transformer. [Diagram 3] FIG. 3 is an explanatory diagram showing buried layer extraction. [Figure 4] FIG. 4 is an explanatory diagram showing an example of a word similarity calculation process. [Diagram 5] FIG. 5 is an explanatory diagram showing an example of a context similarity calculation process. [Figure 6] FIG. 6 is an explanatory diagram illustrating an example of a similarity integration process. [Figure 7] FIG. 7 is an explanatory diagram showing a first processing example of synonym extraction by the synonym extraction control. [Figure 8] FIG. 8 is an explanatory diagram showing an example of a process of synonym extraction using each embedding layer of a plurality of language models by synonym candidate pooling. [Figure 9] FIG. 9 is an explanatory diagram showing an example of processing the synonym candidate ensemble using each embedding layer of a plurality of language models based on the synonym candidate ensemble. [Figure 10] FIG. 10 is an explanatory diagram showing a second processing example of synonym extraction using the synonym extraction control. [Figure 11] FIG. 11 is an explanatory diagram showing an example of a language model selection process by language model selection. [Figure 12] FIG. 12 is an explanatory diagram showing an example of the synonym extraction screen. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] In this embodiment, an embodiment of a synonym extracting device that extracts synonyms of a word in a sentence will be described.

[0014] <Configuration of the synonym extraction device 101> 1 is a block diagram showing an example of the configuration of a synonym extraction device 101 according to this embodiment. The synonym extraction device 101 is configured as a computer including an input device 102, an output device 103, a communication device 104, a calculation device 105, a memory 106, and an auxiliary storage device 107.

[0015] The input device 102 receives inputs of commands, text, voice, and other sensor data from a user, and is, for example, a keyboard, a mouse, a touch panel, a microphone, or other devices.

[0016] The output device 103 is a display device including a display that displays a screen using a liquid crystal, an organic electroluminescence (EL), or a CRT, a printer that prints out on paper, and a speaker that outputs sound. Note that instead of providing the input device 102 and the output device 103, other computers connected via a network may function as the input device 102 and the output device 103.

[0017] The communication device 104 transmits the contents of the processing performed by the synonym extraction device 101 to an external device and receives information from an external device, for example, in accordance with a predetermined protocol.

[0018] The arithmetic device 105 includes a processor and executes a program stored in the memory 106. The memory 106 includes a ROM, which is a non-volatile storage element, and a RAM, which is a non-volatile storage element. The ROM stores an immutable program (e.g., BIOS) and the like. The RAM is a high-speed, volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores the program executed by the arithmetic device 105 and data used when the program is executed.

[0019] The auxiliary storage device 107 is a large-capacity non-volatile storage device constituted by, for example, a magnetic storage device (HDD), a flash memory (SSD), or the like, and stores the programs executed by the arithmetic device 105 and data used when the programs are executed. That is, the programs are read from the auxiliary storage device 107, loaded into the memory 106, and executed by the arithmetic device 105.

[0020] The programs executed by the arithmetic unit 105 are provided to the synonym extraction device 101 from removable media (such as a CD-ROM or a flash memory) or via a network, and are stored in a non-volatile auxiliary storage device 107, which is a non-transitory storage medium. For this reason, it is preferable that the synonym extraction device 101 has an interface for reading data from removable media.

[0021] The synonym extraction device 101 is a computer system that is physically configured on one computer, or is configured on multiple logically or physically configured computers, and may process in separate threads on the same computer, or may process on a virtual computer built on multiple physical computer resources.

[0022] The arithmetic unit 105 functions as the synonym extraction device 101 by executing processes according to the programs loaded into the memory 106 .

[0023] The language model 115 is a language model trained by deep learning, and there may be a plurality of the language models. The embedding layer 116 is an embedding layer extracted from the language model 115, and there may be a plurality of the language models. The language model 115 and the embedding layer 116 may be stored in another computer accessible by the synonym extraction device 101 and the communication device 104.

[0024] The embedding layer extraction 108 is a software module that causes the calculation device 105 to execute a process of extracting the embedding layer 116 from the language model 115 and storing it in the auxiliary storage device 107 .

[0025] The word similarity calculation 109 is a software module that causes the computing device 105 to execute a process of calculating a context-independent similarity between a target word and a synonym candidate using an arbitrary embedding layer 116. This similarity is called a word similarity.

[0026] The context similarity calculation 110 is a software module that causes the arithmetic device 105 to execute a process of calculating the similarity between a target sentence and a replacement sentence in which a target word in the target sentence is replaced with a synonym, using an arbitrary language model 115. This similarity is called a context similarity.

[0027] The similarity integration 111 is a software module that causes the calculation device 105 to execute a process of calculating an integrated similarity that integrates a word similarity and a context similarity.

[0028] The synonym extraction control 117 is a software module that causes the calculation device 105 to execute a process of extracting synonyms for a target word.

[0029] Specifically, for example, the arithmetic unit 105 first calculates the word similarity between all synonym candidates and the target word, selects synonym candidates with high word similarity, and calculates the context similarity for each selected synonym candidate through the synonym extraction control 117. Because the cost of calculating the context similarity is high, the synonym candidates to be used for calculating the context similarity are narrowed down by the word similarity.

[0030] Next, the arithmetic unit 105 calculates an integrated similarity by integrating the word similarity and the context similarity through the synonym extraction control 117, and extracts as synonyms those synonym candidates whose integrated similarity is equal to or greater than a threshold value Ta, or those whose integrated similarities are within the top Na rank (Na is a preset integer equal to or greater than 1). Hereinafter, such synonym candidates will be referred to as synonym candidates with high integrated similarity.

[0031] The synonym candidate pooling 112 is a software module that causes the computing device 105 to execute a process of collecting synonym candidates using multiple embedding layers 116.

[0032] The synonym candidate ensemble 113 is a software module that causes the calculation device 105 to execute a process of calculating a plurality of word similarities using a plurality of embedding layers 116 and ensembling (averaging) these word similarities.

[0033] The synonym extraction control 117 is also a software module that causes the calculation device 105 to execute a process of extracting synonyms for a target word using multiple embedding layers 116.

[0034] Specifically, for example, the arithmetic unit 105 calculates the word similarity in the synonym candidate ensemble 113 for the synonym candidates collected in the synonym candidate pooling 112 by the synonym extraction control 117, and calculates the context similarity for the synonym candidates whose word similarity is equal to or greater than the threshold value Tb, or for the synonym candidates whose word similarity is in the top Nbth place (Nb is a preset integer equal to or greater than 1). Hereinafter, such synonym candidates are referred to as synonym candidates with high word similarity.

[0035] Then, the arithmetic unit 105 extracts, as synonyms, synonym candidates whose integrated similarity obtained by integrating the word similarity and the context similarity is equal to or greater than a threshold value Tc, or synonym candidates whose integrated similarities are in the top Nc order (Nc is a preset integer equal to or greater than 1), through the synonym extraction control 117. Hereinafter, such synonym candidates will be referred to as synonym candidates with high integrated similarity.

[0036] The language model selection 114 is a software module that causes the computing device 105 to automatically select one or more embedding layers 116 to be used in the word similarity calculation 109, the synonym candidate pooling 112, the synonym candidate pooling 112, and the synonym candidate ensemble 113, and to automatically select a language model 115 to be used in the context similarity calculation 110. .

[0037] <Language Model 115> Before describing the details of synonym extraction, the language model 115 and the embedding layer 116 will be described in detail with reference to FIG.

[0038] 2 is a schematic diagram of a language model 115 called a transformer. The language model 115 is a model trained using a first training data set in which a set of sentences including a target word is used as first training data and a synonym of the target word in each sentence is used as first correct answer data. Synonyms of the target word are specified, for example, by a synonym dictionary (not shown) in the auxiliary storage device 107. This first training data set is held in the language model 115. During synonym extraction (prediction), when a target word is selected, words other than the target word in the set of sentences are used as synonym candidates.

[0039] Specifically, for example, a sentence 200 is input to the language model 115. Fig. 2 shows an example of inputting the sentence 200, "I moved, but I did it. Dou shou yoi." In reality, the input sentence 200 is divided into tokens by a program called a tokenizer, and a token string 201 is input to the language model 115. In Fig. 2, a token string 201 consisting of seven tokens, "I moved," "I did it ...

[0040] Each token in the token sequence 201 is converted into a vector of several hundred dimensions in the input embedding layer 202. This vector is called an embedded representation. The input embedding layer 202 is a matrix consisting of the number of token types (vocab_size) and the number of dimensions of the embedded representation (768), and the values ​​of the matrix are learned from a second training dataset (a combination of second training data, which is an existing token sequence 201 obtained from the first training data, and an embedded representation 203 (second correct answer data) obtained from the second training data). The second training dataset is held in the input embedding layer 202.

[0041] The embedding representation 203 is a context-independent embedding representation. In other words, “moving” will have the same embedding representation no matter what context it appears in. In this embodiment, the input embedding layer 202 is used as the embedding layer 116.

[0042] 3 is an explanatory diagram showing the embedding layer extraction 108. That is, the embedding layer extraction 108 executes a process of separating the input embedding layer 202 from the given language model 115 as the embedding layer 116 and storing it.

[0043] Returning to Figure 2, the language model 115 performs an addition process (204) of a position index and a sentence index on each embedded representation 203 that is the output of the input embedding layer 202, and outputs an addition result 205 for each token. The position index is a real value that indicates the appearance position from the beginning of the input token sequence 201, and the sentence index is a real value that distinguishes whether the target token in the token sequence 201 is a token that appears in the first sentence or the second sentence. Each of the addition results 205 is an embedded representation of 768 dimensions.

[0044] Next, the language model 115 converts the 768-dimensional embedding representations, which are each of the addition results 205, into the same 768-dimensional embedding representation 207 in the transformer layer 206. However, the language model 115 calculates the relevance with tokens other than the target token by a mechanism called attention, and then calculates the output 768-dimensional value as the embedding representation 207. The parameters of the transformer layer 206 are also learned from a third training dataset (a combination of the third training data, which is the addition result 205 obtained from the second super-correct data, and the embedding representation 207 (third super-correct data) obtained from the third training data). The third training dataset is held in the transformer layer 206.

[0045] The embedding 207 output by the transformer layer 206 is context-dependent, unlike the embedding 203. In other words, even for the same word "moving," the embedding 207 will be different depending on the context in which it appears.

[0046] The language model 115 repeats the conversion to the embedded representation 207 by the transformer layer 206 one or more times, and outputs the embedded representation 210.

[0047] The output embedding layer 211 converts the top-level embedding representation 210 into a dimension equal to the number of token types (vocab_size), and outputs a real vector 212 equal to the number of token types. The real vector 212 represents an embedding representation of a word that matches the context of the input sentence 200. When training the language model 115, the language model 115 inputs a masked version of "moving" in the token sequence 201, and updates all parameters by backpropagation so that the value of the dimension corresponding to the masked "moving" in the real vector 212 becomes the maximum value.

[0048] In other words, language model 115 defines a set of synonym candidates, and is a model trained to select and output a synonym of the target word from the set of synonym candidates when a sentence including a target word is input.

[0049] When extracting synonyms of "moving", the technology of Non-Patent Document 2 sorts the values ​​of the output vector corresponding to "moving" and sets the tokens corresponding to the higher index as synonyms of "moving". However, because the context-dependent embedding expression 210 is used as the embedding expression for "moving", it is too strongly influenced by the context. In this example, the value of "marriage", which is not synonymous with "moving" but matches well with the context "I did x, but what should I do?" (x is a mask), becomes large.

[0050] In this embodiment, before taking into account the influence of context, tokens whose similarity to the target word “moving” itself is equal to or greater than a threshold value Tb, or tokens whose similarity is within the top Nb levels, are determined as candidates for synonyms with high word similarity.

[0051] <Word similarity calculation 109> 4 is an explanatory diagram showing an example of the processing of the word similarity calculation 109. The tokenize 403, the embedded expression extraction 406, the embedded expression average 409, and the embedded expression similarity calculation 412 are software modules in the word similarity calculation 109.

[0052] The calculation device 105 uses the word similarity calculation 109 to calculate the word similarity between the target word w (401) and the synonym candidate s (402).

[0053] First, the calculation device 105 divides the target word w (401) and the synonym candidate s (402) into tokens by tokenization 403. A token sequence 404 is tokens wt1, wt2, ... obtained as a result of tokenizing the target word w (401). A token sequence 405 is tokens st1, st2, ... obtained as a result of tokenizing the synonym candidate s (402). For example, if the target word w is "owner", the token sequence 404 obtained as a result of tokenizing this target word w is wt1="possession", wt2="person".

[0054] Next, the arithmetic unit 105 extracts the embedded representation of each token in the token sequence 404 and the token sequence 405 by the embedded representation extraction 406, and outputs the embedded representation sequence 407 and the embedded representation sequence 408. The arithmetic unit 105 uses the embedding layer 116 when extracting the embedded representation sequences 407 and 408. For example, the embedded representation of the token wt1 is emb(wt1). The embedded representation is a real vector with several hundred dimensions. When the input embedding layer 202 extracted from the language model 115 in FIG. 2 is used, the number of dimensions of the embedded representation is 768.

[0055] Next, the calculation device 105 averages the embedded expression string 407 extracted from the target word s (401) using the embedded expression average 409, and outputs a single embedded expression w_emb(w) (410). Similarly, the calculation device 105 outputs an embedded expression w_emb(s) (411) for the synonym candidate s (402).

[0056] Finally, the calculation device 105 performs an embedded expression similarity calculation 412 to calculate the cosine (cos(w_emb(w), w_emb(s)) of the embedded expression w_emb(w) (410) for the target word w (401) and the embedded expression w_emb(s) (411) for the synonym candidate s (402), and outputs the word similarity w_sim(w,s) (413).

[0057] The word similarity w_sim(w,s) between a target word w and a synonym candidate s does not take into account the context in which the target word w appears. Therefore, in this embodiment, the calculation device 105 takes into account the context dependency of the synonym by calculating the similarity between a target sentence in which the target word w appears and a synonym sentence in which the target word w in the target sentence is replaced with the synonym candidate s. This similarity is the context similarity.

[0058] <Context similarity calculation 110> 5 is an explanatory diagram showing an example of the processing of the context similarity calculation 110. A word replacement 504, a tokenization 506, a context embedded expression calculation 509, a context embedded expression average 512, and an embedded expression similarity calculation 515 are software modules in the context similarity calculation 110.

[0059] The calculation device 105 uses the context similarity calculation 110 to calculate the context similarity between the target word w (401) and the synonym candidate s (402) for the target sentence c (501).

[0060] First, the calculation device 105 replaces the target word w (401) in the target sentence c (501) with the synonym candidate s (402) by word replacement 504 to create a synonym sentence cs (505). For example, if the target sentence c (501) is "I moved, but what should I do?", the target word w (401) is "I moved," and the synonym candidate s (402) is "I moved," the synonym sentence cs (505) becomes "I moved, but what should I do?". Thereafter, the calculation device 105 calculates the similarity between the target sentence c (501) and the synonym sentence cs (505).

[0061] The arithmetic unit 105 divides the target sentence c (501) and the synonym sentence cs (505) into tokens by tokenizing 506, and outputs token strings 507 and 508. The token string 507 is an array of tokens wt1, wt2, . . ., and the token string 508 is an array of tokens st1, st2, . . .

[0062] Next, the arithmetic unit 105 extracts the embedded expressions of each token in the token sequence 507 and the token sequence 508 by the context embedded expression calculation 509, and obtains the embedded expression sequences 510 and 511. The language model 115 is used when extracting the embedded expressions. The embedded expression sequence 510 is an array of the embedded expressions emb(wt1), emb(wt2), ..., and the embedded expression sequence 511 is an array of the embedded expressions emb(st1), emb(st2), .... The embedded expression sequences 510 and 511 are arrays of the top embedded expressions 210 in FIG. 2, and are context-dependent embedded expressions. For example, the embedded expression of the token wt1 for the target sentence c (501) is emb(wt1|c).

[0063] Next, the calculation device 105 averages the embedded expression strings 510 extracted from the target word context (501) using the context embedded expression averaging 512, and outputs a single embedded expression c_emb(c) (513). Similarly, the calculation device 105 uses the context embedded expression averaging 512 to output an embedded expression c_emb(cs) (514) for the synonym sentence cs (505).

[0064] Finally, the calculation device 105 performs an embedded expression similarity calculation 515 to calculate the cosine (cos(c_emb(c), c_emb(cs)) of the embedded expression c_emb(c) (513) for the target sentence c (501) and the embedded expression c_emb(cs) (514) for the synonym sentence cs (505), and outputs the word similarity c_sim(w, s) (516).

[0065] <Similarity Integration 111> Fig. 6 is an explanatory diagram showing an example of the processing of the similarity integration 111. The calculation device 105 integrates the word similarity 413 calculated in Fig. 4 and the context similarity 516 calculated in Fig. 5 by the similarity integration 111, and outputs an integrated similarity 600. A synonym candidate s having a larger integrated similarity 600 with respect to a target word w is determined to be more likely to be a synonym.

[0066] The calculation device 105 integrates the word similarity w_sim(w,s) (413) and the context similarity c_sim(w,s) (516) by the similarity integration 111 to calculate the integrated similarity wc_sim(w,s) (600). In this embodiment, the calculation device 105 adds the word similarity 413 and the context similarity 516 using a coefficient α. Specifically, for example, the integrated similarity 600 is calculated by the following formula (1).

[0067] wc_sim(w,s)=(1-α)*w_sim(w,s)+α*c_sim(w,s)···(1)

[0068] <Synonym extraction> Fig. 7 is an explanatory diagram showing a first processing example of synonym extraction by the synonym extraction control 117. Fig. 7 shows an example of extracting synonyms by combining the word similarity calculation 109 (Fig. 4), the context similarity calculation 110 (Fig. 5), and the similarity integration 111 (Fig. 6).

[0069] The input data is a target word 401 and a target sentence 501 in which the target word 401 appears. Also, an embedding layer 116 for calculating the word similarity 413 and a language model 115 for calculating the context similarity 516 are specified in advance. Note that these embedding layer 116 and language model 115 may be selected arbitrarily.

[0070] First, the arithmetic unit 105 obtains a synonym candidate set 703 by the synonym extraction control 117. In Fig. 7, a set of tokens defined in the embedding layer 116 is set as the synonym candidate set 703, but any set of character strings may be set as the synonym candidate set. For example, the arithmetic unit 105 may use an existing dictionary such as wordnet to extract synonyms of the target word 401 and set these as the synonym candidate set 703. Alternatively, the arithmetic unit 105 may add the synonym candidate 402 extracted from an existing dictionary to the synonym candidate set 703 extracted from the embedding layer 116.

[0071] The arithmetic unit 105 executes a word similarity calculation process for each synonym candidate in the synonym candidate set 703 by the word similarity calculation 109 , and outputs a word similarity set 705 which is a set of the word similarities 413 .

[0072] The arithmetic unit 105 sorts the word similarity set 705 in descending order of the word similarity 413 by the word similarity sorting / selection 706, selects the top arbitrary number Nb or target words 401 equal to or greater than the threshold value Tb, and outputs a new word similarity set 707. The word similarity set 707 includes synonym candidates 402 with high word similarities 413.

[0073] The arithmetic unit 105 executes a context similarity calculation process using the context similarity calculation 110 for the synonym candidates 402 with high word similarities 413 in the word similarity set 707 to calculate the context similarities 516 and output the context similarity set 709 .

[0074] The calculation cost of the context similarity 516 is higher than the calculation cost of the word similarity 413. Specifically, this is because when calculating the word similarity 413, an embedded expression is obtained from one layer of the embedding layer 116, whereas when calculating the context similarity 516, an embedded expression is obtained using multiple layers of the language model 115. Therefore, in this embodiment, as an example, the context similarity 516 is calculated only for synonym candidates 402 with high word similarity 413. However, this does not prevent the context similarity 516 from being calculated for synonym candidates 402 with not high word similarity 413.

[0075] The arithmetic unit 105 executes a similarity integration process by the similarity integration 111 to integrate the word similarity set 707 and the context similarity set 709 and output an integrated similarity set 711 .

[0076] Finally, the calculation device 105 sorts the integrated similarity set 711 in descending order of the integrated similarity 600 by integrated similarity sorting / selection 712, selects the top arbitrary number Nc or ​​synonym candidates 402 equal to or greater than a threshold value Tc, and outputs a new integrated similarity set 713. The synonym candidates 402 in the integrated similarity set 713 (synonym candidates with high integrated similarity) are synonyms of the target word 401.

[0077] So far, we have described synonym extraction using a single embedding layer, but from here on, we will describe a synonym extraction method using multiple embedding layers.

[0078] 8 is an explanatory diagram showing a processing example of synonym extraction using each embedding layer of a plurality of language models by the synonym candidate pooling 112. First, the calculation device 105 collects synonym candidate sets (first synonym candidate set 802-1, ..., i-th synonym candidate set 802-i, ..., n-th synonym candidate set 802-n) from each of a plurality of embedding layers, namely, a first embedding layer 801-1, ..., an i-th embedding layer 801-i, ..., an n-th embedding layer 801-n (i is an integer satisfying 1≦i≦n, n is an integer equal to or greater than 2, and corresponds to the input embedding layer 202 in FIG. 2).

[0079] In Fig. 8, when a target word 401 is input, a set of tokens defined in the ith embedding layer 801-i is set as the ith synonym candidate set 802-i. Next, the calculation device 105 calculates the word similarity 413 for the target word 401 and each synonym candidate in the ith synonym candidate set 802-i using the similarity calculation process described in Fig. 4. The calculation device 105 sorts these word similarities 413 in descending order, selects any number of top word similarities 413 or those equal to or greater than a threshold, and outputs the ith word similarity set 804-i. The above process is the ith word similarity calculation / sorting / selection 803-i.

[0080] The arithmetic unit 105 executes the i-th word similarity calculation / sorting / selection 803-i for the i-th embedded layer 801-i, and outputs a first synonym candidate set 802-1, ..., an i-th synonym candidate set 802-i, ..., an n-th synonym candidate set 802-n.

[0081] In the pooling 800, the calculation device 105 calculates the union of n word similarity sets (first word similarity set 804-1, ..., ith word similarity set 804-i, ..., nth word similarity set 804-n) and outputs a new word similarity set 810. A set of synonym candidates corresponding to each word similarity in this word similarity set 810 becomes a synonym candidate set extracted from the first embedded layer 801-1, ..., ith embedded layer 801-i, ..., nth embedded layer 801-n (hereinafter referred to as synonym candidate set 810).

[0082] By extracting synonym candidates from the first embedded layer 801-1, ..., the i-th embedded layer 801-i, ..., the n-th embedded layer 801-n, it becomes possible to extract various synonym candidates depending on the tokenizer assumed by each i-th embedded layer 801-i.

[0083] For example, from the i-th embedded layer 801-i, which is based on a word dictionary-based tokenizer, it is not possible to extract "Owner" as a synonym candidate of "Owner". This is because "Owner" is usually split into two tokens, "Owner" and "Person". On the other hand, with a statistics-based tokenizer, if "Owner" appears frequently in the corpus, it may be possible to extract "Owner" as a single token. Therefore, by adding the i-th embedded layer 801-i, which uses such a tokenizer, it is possible to extract "Owner" as a synonym candidate of "Owner".

[0084] For the set of synonym candidates 810 extracted from the first embedded layer 801-1, ..., the i-th embedded layer 801-i, ..., the n-th embedded layer 801-n, the arithmetic unit 105 calculates the word similarity 413 using the first embedded layer 801-1, ..., the i-th embedded layer 801-i, ..., the n-th embedded layer 801-n.

[0085] FIG. 9 is an explanatory diagram showing an example of synonym candidate ensemble processing using each embedding layer of a plurality of language models by the synonym candidate ensemble 113. The synonym candidate ensemble 113 is a result of calculating the word similarity between the target word w (401) and the synonym candidate s (402) for each language model and ensembling it. The synonym candidate ensemble 113 is a result of execution for each synonym candidate s (402). In FIG. 9, the target word 401, the synonym candidate s (402), and the first embedding layer 801-1, ..., the i-th embedding layer 801-i, ..., the n-th embedding layer 801-n are given. The synonym candidate s (402) is each element of the synonym candidate set 810 extracted in FIG. 8. In addition, the n embedding layers in FIG. 9 are the same as the n embedding layers in FIG. 8, but may be different.

[0086] For the i-th embedded layer 801-i, the arithmetic device 105 calculates the word similarity w_simi(w,s) (901-i) for the target word w (401) and the synonym candidate s (402) using the word similarity calculation process described in Figure 4 in the word similarity calculation 900-i.

[0087] In the ensemble 910, the calculation device 105 averages the calculated N word similarities 901-1, . . . , 901-i, . . . , 901-n to output an ensemble word similarity w_sim(w,s) (920).

[0088] The word similarity 901-i may be accurate or inaccurate depending on the i-th embedding layer 801-i used. In this embodiment, by ensembling the word similarities 901-1, ..., 901-i, ..., 901-n calculated from the first embedding layer 801-1, ..., the i-th embedding layer 801-i, ..., the n-th embedding layer 801-n, the dependency of the word similarity 901-i on each individual i-th embedding layer 801-i can be reduced.

[0089] Fig. 10 is an explanatory diagram showing a second processing example of synonym extraction by the synonym extraction control 117. Fig. 10 shows a process of extracting synonyms by applying the synonym candidate pooling 112 (Fig. 8) and the synonym candidate ensemble 113 (Fig. 9).

[0090] The input data is a target word 401 and a target sentence 501 in which the target word 401 appears. In addition, multiple embedding layers 116 used in the synonym candidate pooling 112 and the synonym candidate ensemble 113, and a language model 115 for calculating the context similarity 516 are also prepared in advance. Note that the multiple embedding layers 116 and the language model 115 may be selected arbitrarily.

[0091] First, the computing device 105 uses the synonym candidate pooling 112 (FIG. 8) to extract a synonym candidate set 810. Next, the computing device 105 uses the synonym candidate ensemble 113 (FIG. 9) to output a word similarity set 1006. The word similarity set 1006 is a set of one or more lexical similarities. The word similarity set 1006 becomes a synonym candidate set. As already mentioned, using multiple embedding layers increases the diversity of synonym candidates and reduces the dependency of the word similarities 901-i on individual i-th embedding layers 801-i.

[0092] In the context similarity calculation 110, the calculation device 105 calculates the context similarity 516 with the target word 401 for each synonym candidate in the synonym candidate set 1006 using the context similarity calculation process described in FIG. 5, and outputs a context similarity set 1008.

[0093] In the similarity integration 111, the calculation device 105 integrates the word similarity set 1006 and the context similarity set 1008 using the similarity integration process described in Fig. 6, and outputs an integrated similarity set 1012. The synonym candidates in this set become the extracted synonyms.

[0094] In both the synonym extraction process of Fig. 7 and the synonym extraction process of Fig. 10, the embedding layer 116 and the language model 115 to be used are selected in advance. It is also possible to automatically select these embedding layer 116 and language model 115. In the case of the embedding layer, the language model from which the embedding layer is extracted is selected.

[0095] Fig. 11 is an explanatory diagram showing an example of a language model selection process by the language model selection 114. Fig. 11 shows an example in which a language model suitable for synonym extraction processing is automatically selected from a given language model set 1102.

[0096] To select a language model, a sentence set 1101 of the target domain for synonym extraction is provided. This sentence set 1101 may be any sentence set of the target domain. In model compatibility calculation 1103, the computing device 105 measures how well each language model in the language model set 1102 to be selected matches the provided sentence set 1101, for example by perplexity, which is a well-known measure.

[0097] In model fitness sorting / selection 1104, the calculation device 105 sorts the language model set 1102 in descending order of model fitness, and selects an arbitrary number of language models from the top or a language model set 1105 having a model fitness equal to or greater than a threshold value.

[0098] 12 is an explanatory diagram showing an example of a synonym extraction screen. A synonym extraction screen 1201 is displayed on the output device 103. The synonym extraction screen 1201 has a target word input area 1202, a context input area 1203, a dictionary list 1204, a language model list 1205, an automatic selection button 1206, a sentence collection file designation button 1207, a synonym candidate extraction button 1208, a synonym candidate list 1209, a save button 1210, an add button 1211, a language model list 1212, an automatic selection button 1213, a sentence collection file designation button 1214, a synonym candidate ranking button 1215, a synonym candidate list 1216, and a save button 1217.

[0099] The subject word input area 1202 is an area where the subject word 401 is input by a user operation. The context input area 1203 is an area where the subject sentence 501 is input by a user operation.

[0100] The dictionary list 1204 lists selectable dictionaries, and the user can select one by using a radio button. The language model list 1205 lists the embedding layer 116 used as a source of synonym candidates, and the user can select one by using a radio button.

[0101] An automatic selection button 1206 is a user interface for automatically selecting a language model in the language model list 1205 when pressed by the user.

[0102] The sentence set file designation button 1207 is a user interface for designating a sentence set by the user's pressing. The user can automatically select a language model (embedding layer) suitable for the designated sentence set by designating a sentence set with the sentence set file designation button 1207 and pressing the automatic selection button 1206. This process is executed in the language model selection process described in FIG. 11.

[0103] An extract synonym candidate button 1208 is a user interface for listing synonym candidates using a dictionary selected in the dictionary list 1204 and a language model (embedded layer) selected in the language model list 1205 when pressed by the user.

[0104] A synonym candidate list 1209 displays a set of synonym candidates 1006 (see FIG. 10) that is listed by pressing the extract synonym candidate button 1208. That is, synonym candidates with large word similarity 901-i are listed. In the synonym candidate list 1209, the word similarity 901-i is displayed as a score. The user can select an appropriate synonym candidate from the listed synonym candidate. The user can also directly edit the character string of the displayed synonym candidate to change it.

[0105] A save button 1210 is a user interface for saving synonyms in the synonym candidate list 1209 in the auxiliary storage device 107 when pressed by the user. An add button 1211 is a user interface for adding a new row to the synonym candidate list 1209 and inputting a new synonym candidate when pressed by the user.

[0106] The language model list 1212 lists language models used in ranking synonym candidates, and the user can select one by using a radio button.

[0107] An automatic selection button 1213 is a user interface for automatically selecting a language model in the language model list 1212 when pressed by the user.

[0108] The sentence set file designation button 1214 is a user interface for designating a sentence set by the user's pressing. The user can automatically select a language model (embedded layer) suitable for the designated sentence set by designating a sentence set with the sentence set file designation button 1214 and pressing the automatic selection button 1213. This process is executed in the language model selection process described in FIG. 11.

[0109] A synonym candidate ranking button 1215 is a user interface for, when pressed by the user, ranking the synonym candidates ranked in the synonym candidate list 1209 using a language model selected in the language model list 1212 .

[0110] In the synonym candidate list 1216, the synonym candidates selected in the synonym candidate list 1209 are displayed in rank order using the language model selected in the language model list 1212.

[0111] The save button 1217 is a user interface for saving the synonyms in the synonym candidate list 1216 in the auxiliary storage device 107 when pressed by the user.

[0112] That is, the calculation device 105 calculates the context similarity for the synonym candidate selected in the synonym candidate list 1209 using the language model selected in the language model list 1212, and sorts the synonym candidates by integrated similarity obtained by integrating the context similarity with the already calculated word similarity. The user can select an appropriate synonym from the list and press the save button 1217 to save the selected synonym in the auxiliary storage device 107.

[0113] As described above, according to the present embodiment, synonym extraction that is balanced between context-independence and context-dependence is possible. In addition, by using multiple language models, synonym extraction can be performed with high accuracy and stability.

[0114] The present invention is not limited to the above-described embodiments, and includes various modified examples and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the configurations described. Also, a part of the configuration of one embodiment may be replaced with a configuration of another embodiment. Also, a configuration of another embodiment may be added to a configuration of one embodiment. Also, a part of the configuration of each embodiment may be added, deleted, or replaced with another configuration.

[0115] In addition, each of the configurations, functions, processing units, processing means, etc. described above may be realized in hardware, part or all of which may be designed, for example, as an integrated circuit, or may be realized in software, by a processor interpreting and executing a program that realizes each function.

[0116] Information such as programs, tables, files, etc. that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC (Integrated Circuit) card, an SD card, or a DVD (Digital Versatile Disc).

[0117] In addition, the control lines and information lines shown are those considered necessary for the explanation, and do not necessarily show all the control lines and information lines necessary for implementation. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0118] 101 Synonym extraction device 105 Arithmetic equipment 108 Embedding Layer Extraction 109 Word similarity calculation 110 Context similarity calculation 111 Similarity Integration 112 Synonym Candidate Pooling 113 Synonym Candidate Ensemble 114 Language Model Selection 115 Language Models 116 Buried Layer 117 Synonym Extraction Control

Claims

1. A synonym extraction device having a processor for executing a program and a storage device for storing the program, the device having access to a language model trained to output synonyms of words in a sentence, the device comprising: the language model has a first layer and a second layer; The first layer converts words constituting the sentence into a first embedding representation that is independent of the context of the sentence; the second layer generates a second embedding representation that depends on a context of the sentence based on the position of the word, the sentence in which the word appears, and the first embedding representation of the word; The processor, a word similarity calculation process in which a word other than the target word in the target sentence is treated as a synonym candidate, a first embedded expression for the target word and a first embedded expression for the synonym candidate are generated using the first layer, and a word similarity, which is a similarity between the target word and the synonym candidate, is calculated based on the first embedded expression for the target word and the first embedded expression for the synonym candidate; a context similarity calculation process for generating a synonym sentence by replacing a target word in the target sentence with the synonym candidate, generating a second embedded expression for the target sentence and a second embedded expression for the synonym sentence using the first layer and the second layer, and calculating a context similarity, which is a similarity between the target sentence and the synonym sentence, based on the second embedded expression for the target sentence and the second embedded expression for the synonym sentence; a synonym extraction process for extracting the synonyms based on the word similarity and the context similarity; A synonym extraction device comprising:

2. 2. The synonym extraction device according to claim 1, In the word similarity calculation process, the processor calculates the word similarity based on an average value of a first embedded expression for each token constituting the target word and an average value of a first embedded expression for each token constituting the synonym candidate. A synonym extraction device comprising:

3. 2. The synonym extraction device according to claim 1, The processor, a synonym candidate pooling process that uses the first layer to generate a first embedding expression for a synonym candidate in each of a plurality of synonym candidate sets, calculates the word similarity for each of the synonym candidates and for each of the synonym candidate sets by the word similarity calculation process, pools the set of word similarities for each of the synonym candidate sets based on a condition related to the word similarity, and outputs a specific synonym candidate set corresponding to the pooled word similarity; a synonym candidate ensemble process for calculating a specific word similarity between the target word and a specific synonym candidate in the specific synonym candidate set by the word similarity calculation process, and ensembling the specific word similarities for each of the specific synonym candidates; In the context similarity calculation process, the processor replaces a target word in the target sentence with an ensemble result of the synonym candidate ensemble process to generate a specific synonym sentence, and calculates a context similarity between the target sentence and the specific synonym sentence. A synonym extraction device comprising:

4. 2. The synonym extraction device according to claim 1, The processor, calculating a suitability of each of the plurality of language models with respect to a set of sentences, and executing a selection process of selecting a specific language model from the plurality of language models based on the suitability; the processor executes the word similarity calculation process and the context similarity calculation process by using the specific language model selected by the selection process. A synonym extraction device comprising:

5. A synonym extraction method executed by a synonym extraction device having a processor for executing a program and a storage device for storing the program, the synonym extraction device having access to a language model trained to output synonyms of words in a sentence, the method comprising: the language model has a first layer and a second layer; The first layer converts words constituting the sentence into a first embedding representation that is independent of the context of the sentence; the second layer generates a second embedding representation that depends on a context of the sentence based on the position of the word, the sentence in which the word appears, and the first embedding representation of the word; The processor, a word similarity calculation process in which a word other than the target word in the target sentence is treated as a synonym candidate, a first embedded expression for the target word and a first embedded expression for the synonym candidate are generated using the first layer, and a word similarity, which is a similarity between the target word and the synonym candidate, is calculated based on the first embedded expression for the target word and the first embedded expression for the synonym candidate; a context similarity calculation process for generating a synonym sentence by replacing a target word in the target sentence with the synonym candidate, generating a second embedded expression for the target sentence and a second embedded expression for the synonym sentence using the first layer and the second layer, and calculating a context similarity, which is a similarity between the target sentence and the synonym sentence, based on the second embedded expression for the target sentence and the second embedded expression for the synonym sentence; a synonym extraction process for extracting the synonyms based on the word similarity and the context similarity; A synonym extraction method comprising the steps of:

6. A synonym extraction program that causes a processor to perform simultaneous word extraction using a language model trained to output synonyms of words in a sentence, the language model has a first layer and a second layer; The first layer converts words constituting the sentence into a first embedding representation that is independent of the context of the sentence; the second layer generates a second embedding representation that depends on a context of the sentence based on the position of the word, the sentence in which the word appears, and the first embedding representation of the word; The processor, a word similarity calculation process in which a word other than the target word in the target sentence is treated as a synonym candidate, a first embedded expression for the target word and a first embedded expression for the synonym candidate are generated using the first layer, and a word similarity, which is a similarity between the target word and the synonym candidate, is calculated based on the first embedded expression for the target word and the first embedded expression for the synonym candidate; a context similarity calculation process for generating a synonym sentence by replacing a target word in the target sentence with the synonym candidate, generating a second embedded expression for the target sentence and a second embedded expression for the synonym sentence using the first layer and the second layer, and calculating a context similarity, which is a similarity between the target sentence and the synonym sentence, based on the second embedded expression for the target sentence and the second embedded expression for the synonym sentence; a synonym extraction process for extracting the synonyms based on the word similarity and the context similarity; A synonym extraction program for executing the above-mentioned steps.