Knowledge determination device, program and method for detecting knowledge errors in sentences

Abstract

To provide a device and the like for detecting a knowledge error in a sentence.SOLUTION: A knowledge determination deice has: a knowledge database that accumulates knowledge in which substantial words are connected to each other with associated words; substantial word extraction means that extracts, from a sentence, substantial words of a predetermined condition; substantial word pair extraction means that extracts a substantial word pair that is the combination of the substantial words; associated word estimation means that estimates associated words in the substantial word pair from the sentence; interaction graph generation means that generates an interaction graph connecting the substantial word pair with the associated words estimated by the associated word estimation means; knowledge graph generation means that, by using the knowledge database, searches for the associated words in the substantial word pair, and generates a knowledge graph connecting the substantial word pair with the associated words; feature quantity generation means that generates a feature quantity for every substantial word from the interaction graph and the knowledge graph respectively; and a classification learning engine that receives input of the feature quantity of the substantial words in the interaction graph and the feature quantity of the substantial words in the knowledge graph, and classifies the feature quantities into knowledge validity or knowledge error.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technique for detecting knowledge errors in sentences, and is particularly suited to a technique for a dialogue server that generates a response sentence in response to a sentence uttered by a user. [Background technology]

[0002] Dialogue servers are expected to use AI (Artificial Intelligence) to respond to user utterances with natural-sounding responses, thereby increasing the user's willingness to engage in dialogue. However, the dialogue server also needs to strengthen its knowledge of the responses it generates to users. In this case, it is important that the responses generated by the dialogue server in response to user utterances are free of knowledge-based errors.

[0003] Conventionally, there is a technology that uses, for example, RoBERTa (Robustly optimized BERT (Bidirectional Encoder Representations from Transformers) approach) as a language model to determine knowledge errors in each word included in a response sentence (see, for example, Non-Patent Document 1). According to this technology, knowledge is represented as a "triple." A triple represents knowledge using a set of three entities: a "head entity" that is the starting point, a "relation" that represents the relationship, and a "tail entity" that is the end point. RoBERTa takes as input the word sequence of a knowledge triple, the word sequence of a user utterance, and the word sequence of a response sentence from the dialogue server, and outputs whether the knowledge is correct (label F) or incorrect (label T) for each word. This technology requires manual annotation in advance to link related knowledge to each response sentence.

[0004] BERT (registered trademark) is a technology that applies a distributed representation generation algorithm (embedding) to replace data with high-dimensional vectors. This is an encoded representation of bidirectional learning using the Transformer architecture, and is a natural language processing model from Google (registered trademark). BERT is also a Seq2seq-based pre-training model that learns by processing unlabeled features (distributed representations) using the Transformer. This not only predicts the next word in a series of sentences, but also predicts words that are masked from the surrounding context in both directions. This allows the system to learn contextual information corresponding to words, enabling natural language processing.

[0005] There is also a technology that extracts knowledge from text and creates a knowledge graph (see, for example, Non-Patent Document 2). This technology uses a pre-trained language model that accumulates a large amount of knowledge to extract knowledge candidates indicated by text and map them to a knowledge graph. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] N. Dziri, A. Madotto, O. Zaiane, and A. Bose. Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding, 2021, Empirical Methods in Natural Language Processing, pp. 2197-2214. [online] [Retrieved September 17, 2022], Internet<URL:https: / / arxiv.org / abs / 2104.08455> [Non-patent document 2] C. Wang, X. Liu, D. Song. Language Models are Open Knowledge Graphs, 2021, [online], [Retrieved September 17, 2022], Internet<URL:https: / / openreview.net / forum?id=aRTRjVPkm-> [Non-patent document 3] OpenNRE, [online], [searched September 17, 2022], Internet<URL:https: / / github.com / thunlp / OpenNRE> [Non-patent document 4] OpenDialKG, [online], [searched October 5, 2022], Internet<URL:https: / / github.com / facebookresearch / opendialkg> Summary of the Invention [Problem to be solved by the invention]

[0007] A dialogue server can respond to a user's utterance with a response sentence containing a lot of knowledge. In response to this, the user generally recognizes that the response sentence generated by the dialogue server is correct. However, it is difficult to determine whether the knowledge contained in the response sentence is correct or incorrect.

[0008] In contrast, the technology described in Non-Patent Document 1 above has the following problems. (A) Because the learning model is a black box, the basis for determining whether a knowledge error exists within the knowledge connections contained in the text is unclear. (B) It requires prior manual annotation of related knowledge (e.g., related terms), which affects the accuracy of detecting knowledge errors.

[0009] Furthermore, the technology described in Non-Patent Document 2 mentioned above has the following problems. (C) It is not possible to detect knowledge errors as words contained in a sentence. (D) Because the dialogue structure is not taken into consideration, unnecessary connections of knowledge are extracted.

[0010] In response to this, the inventors of the present application wondered whether it would be possible to detect word knowledge errors in sentences (e.g., response sentences to sentences spoken by a user) without requiring annotation of related knowledge, and to clarify the basis for such judgments.

[0011] Therefore, an object of the present invention is to provide a knowledge determination device, program, and method that can detect knowledge errors in words in a sentence without requiring annotation of related knowledge, and can clarify the basis for the determination. [Means for solving the problem]

[0012] According to the present invention, in a knowledge determination device for detecting knowledge errors in a sentence, A knowledge database that stores knowledge that links entity words with relational words, a substance word extraction means for extracting substance words that satisfy a predetermined condition from a sentence; an entity word pair extraction means for extracting entity word pairs each of which is a combination of entity words; a relative word inference means for inferring a relative word in the entity word pair from the sentence; a dialogue graph generating means for generating a dialogue graph in which entity word pairs are connected by the relational words estimated by the relational word estimation means; a knowledge graph generating means for searching for related terms in entity word pairs using a knowledge database and generating a knowledge graph connecting entity word pairs with the related terms; a feature generating means for generating a feature for each entity word from each of the dialogue graph and the knowledge graph; a classification learning engine that inputs, for each entity word, the feature value of the entity word in the dialogue graph and the feature value of the entity word in the knowledge graph and classifies the knowledge as correct or incorrect; The present invention is characterized by having the following.

[0013] According to another embodiment of the knowledge determination device of the present invention, To detect knowledge errors in responses to user utterances, The entity word extraction means extracts entity words that satisfy a predetermined condition from the utterance sentence and the response sentence; the entity word pair extraction means extracts pairs of entity words of the utterance sentence and entity words of the response sentence, and / or entity word pairs that are combinations of entity words of the response sentence; The classification learning engine classifies the entity words contained in the response sentence. It is also preferable.

[0014] According to another embodiment of the knowledge determination device of the present invention, The entity word extraction means extracts entity words of named entities and / or pre-registered entity words as a predetermined condition. It is also preferable.

[0015] According to another embodiment of the knowledge determination device of the present invention, A dialogue graph display means is provided for visually displaying the dialogue graph and highlighting the nodes of entity words classified as knowledge errors. It is also preferable to have

[0016] According to another embodiment of the knowledge determination device of the present invention, The related word estimation means is a dialogue relational word inference engine for inferring relational words in entity word pairs from the sentence; an entity word pair deletion function for deleting entity word pairs for which related words have not been estimated by the dialogue related word estimation engine from among the entity word pairs extracted by the entity word pair extraction means; It is also preferred to include:

[0017] According to another embodiment of the knowledge determination device of the present invention, The dialogue graph generating means generates a dialogue graph in which entity word pairs are connected by the relational words estimated by the dialogue relational word estimation engine of the relational word estimation means, The knowledge graph generating means uses the knowledge database to search for related terms for entity word pairs that have not been deleted by the entity word pair deletion function of the related term inference means, and generates a knowledge graph that connects entity word pairs with the related terms. It is also preferable.

[0018] According to another embodiment of the knowledge determination device of the present invention, The knowledge database is a collection of correct knowledge expressed in triples, which link entity words with relational terms. It is also preferable.

[0019] According to another embodiment of the knowledge determination device of the present invention, The feature generation method is a GNN (Graph Neural Network), The classification learning engine is a Feed Forward Neural network (FFN). It is also preferable.

[0020] According to another embodiment of the knowledge determination device of the present invention, To pre-train the classification learning engine, Instead of the sentences, a dialogue database is provided that stores utterances and responses based on correct knowledge, a positive example graph generating means for generating a positive example graph by connecting a predetermined range of entity word pairs extracted by the entity word extraction means and the entity word pair extraction means from the utterance sentences and the response sentences of the dialogue database with the related word inferred by the related word inferring means; a negative example graph generating means for generating a negative example graph by replacing any entity word with another entity word and / or replacing any related word with another related word or deleting the related word from the positive example graph; and The feature generating means generates a feature for each entity word from each of the positive example graph and the negative example graph, In the training stage, the classification learning engine is trained to input the feature quantities of the positive example graph and the feature quantities of the negative example graph for each entity word, and to output a label indicating correct knowledge if the entity word is not substituted as the negative example graph, and to output a label indicating incorrect knowledge if the entity word is substituted. It is also preferable.

[0021] According to the present invention, a program for causing a computer installed in a device for detecting knowledge errors in a sentence to function includes: A knowledge database that stores knowledge that links entity words with relational words, a substance word extraction means for extracting substance words that satisfy a predetermined condition from a sentence; an entity word pair extraction means for extracting entity word pairs each of which is a combination of entity words; a relative word inference means for inferring a relative word in the entity word pair from the sentence; a dialogue graph generating means for generating a dialogue graph in which entity word pairs are connected by the relational words estimated by the relational word estimation means; a knowledge graph generating means for searching for related terms in entity word pairs using a knowledge database and generating a knowledge graph connecting entity word pairs with the related terms; a feature generating means for generating a feature for each entity word from each of the dialogue graph and the knowledge graph; a classification learning engine that inputs, for each entity word, the feature value of the entity word in the dialogue graph and the feature value of the entity word in the knowledge graph and classifies the knowledge as correct or incorrect; The present invention is characterized by the fact that it makes a computer function by using the above-mentioned method.

[0022] According to the present invention, in a knowledge determination method for a device for detecting knowledge errors in a sentence, The device has a knowledge database that stores knowledge that links entity words with relational words, A first step of extracting entity words that satisfy predetermined conditions from a sentence; a second step of extracting entity word pairs, which are combinations of entity words; a third step of estimating the relative terms in the entity word pair from the sentence; a fourth step of generating a dialogue graph connecting the entity word pairs with the relative terms estimated in the third step, and searching for the relative terms in the entity word pairs using a knowledge database to generate a knowledge graph connecting the entity word pairs with the relative terms; a fifth step of generating features for each entity word from the dialogue graph and the knowledge graph; a sixth step of inputting, for each entity word, the feature quantity of the entity word in the dialogue graph and the feature quantity of the entity word in the knowledge graph using a classification learning engine, and classifying the knowledge as correct or incorrect; The present invention is characterized by carrying out the following. [Effects of the Invention]

[0023] According to the knowledge determination device, program, and method of the present invention, it is possible to detect knowledge errors in words in a sentence and clarify the basis for the determination without requiring annotation of related knowledge. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 1 is a diagram illustrating a system configuration for realizing a dialogue with a user. [Figure 2] 1 is a functional configuration diagram of a knowledge determination device according to the present invention; [Figure 3] FIG. 2 is an explanatory diagram of an entity word extraction unit and an entity word pair extraction unit in the present invention. [Figure 4] FIG. 2 is an explanatory diagram of a related word inference unit in the present invention. [Figure 5] 3 is an explanatory diagram of a dialogue graph generation unit and a knowledge graph generation unit in the present invention. FIG. [Figure 6] FIG. 2 is an explanatory diagram of a feature generator and a classification learning engine according to the present invention. [Figure 7] FIG. 2 is a functional configuration diagram of a knowledge determination device in a training stage according to the present invention. [Figure 8] FIG. 1 is an explanatory diagram illustrating training of a classification learning engine by a positive example graph generation unit and a negative example graph generation unit in the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0025] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0026] FIG. 1 is a diagram showing the system configuration for realizing a dialogue with a user.

[0027] The system of FIG. 1 includes a knowledge determination device 1, a terminal 2, and a dialogue server 3. The terminal 2 transmits the user's utterance to the dialogue server 3. The dialogue server 3 generates a response sentence that will result in a natural dialogue in response to the utterance, and returns the response sentence to the terminal 2. The terminal 2 then replies to the user with the response sentence by voice or text. The dialogue server 3 accumulates utterances and responses from past dialogues with the user as a dialogue history.

[0028] As a user interface, the terminal 2 has a speech recognition function for the user's speech and a speech synthesis function for generating a response to the user. The speech recognition function converts the user's speech, captured by the microphone of the terminal 2, into a text-based speech. The speech synthesis function converts the response generated by the dialogue server 3 into a voice signal. Of course, the voice recognition function and voice synthesis function may be installed in the terminal 2 or in the dialogue server 3.

[0029] According to FIG. 1, the dialogue server 3 stores, for example, the following dialogue history between the user and the dialogue server 3. User utterance: Could you recommend a couple of books by Richard Adams? Response from the dialogue server: Sure! He wrote Watership Down in 1991. It is speculative fiction.

[0030] The knowledge determination device 1 of the present invention detects knowledge errors in "sentences" in the simplest terms. 1, the knowledge determination device 1 may detect a knowledge error in a response sentence generated by the dialogue server 3 in response to a user's utterance sentence. The knowledge determination device 1 acquires a pair of an utterance sentence and a response sentence from the dialogue server 3 as a dialogue history. The response sentence determined by the knowledge determination device 1 is not limited to one generated by the dialogue server 3, but may be one responded to by another user. In other words, the response sentence may be one that is in response to a sentence uttered by a user, and it does not matter who generated the response sentence.

[0031] FIG. 2 is a functional configuration diagram of the knowledge determination device according to the present invention.

[0032] According to FIG. 2, the knowledge determination device 1 includes a dialogue history storage unit 100 and a knowledge database 101.

[0033] [Dialogue history storage unit 100] The dialogue history storage unit 100 stores a series of dialogue histories that have been exchanged in large quantities in the past. The dialogue history includes a plurality of pairs of utterance sentences and response sentences. Of course, the dialogue history storage unit 100 does not need to be stored in the knowledge determination device 1 itself, but it does need to be input from outside at the time of determination.

[0034] In another embodiment, the knowledge determination device 1 may detect knowledge errors from real-time utterances and response sentences. In this case, a user's utterance and a response sentence intended to respond to the user are input to the knowledge determination device 1 in real time.

[0035] [Knowledge Database 101] The knowledge database 101 stores knowledge in which entities are linked together by relational terms. The knowledge stored in the knowledge database 101 may be expressed in the form of a "triple" in which entity words are linked by relational terms. As mentioned above, a triple represents knowledge using a set of three entities: a starting entity word (head entity), a relation, and an ending entity word (tail entity).

[0036] The knowledge database 101 may be, for example, a large-scale collection of triples constructed from Wikipedia (registered trademark). Words described in Wikipedia are stored as entities, and related terms connecting words are stored as relations. In other words, the knowledge database 101 is a publicly available dataset that has been manually compiled and is assumed to be correct as knowledge.

[0037] The knowledge determination device 1 also includes an entity word extraction unit 11, an entity word pair extraction unit 12, a related word estimation unit 13, a dialogue graph generation unit 141, a knowledge graph generation unit 142, a feature generation unit 15, a classification learning engine 16, and a dialogue graph display unit 17. These functional components are realized by executing a program that causes a computer installed in the device to function. The processing flow of these functional components can also be understood as a knowledge determination method.

[0038] FIG. 3 is an explanatory diagram of the entity word extraction unit and entity word pair extraction unit in the present invention.

[0039] [Entity Word Extraction Unit 11] The substantive word extraction unit 11 extracts substantive words that satisfy predetermined conditions from the "sentence." The entity word extraction unit 11 inputs, as sentences, a series of utterances and responses stored in the dialogue history storage unit 100. Of course, in another embodiment, only "sentences" for which knowledge errors are to be detected may be input.

[0040] In this case, it is difficult to determine what kind of entity words should be extracted. If all words are treated as entity words, there is a risk that a large number of unnecessary triples that cannot be considered knowledge will be extracted, leading to a decrease in judgment accuracy.

[0041] For this purpose, the entity word extraction unit 11 extracts entity words that satisfy a predetermined condition. Here, it is preferable that the "predetermined condition" be entity words of named entities and / or pre-registered entity words. Named entities are particularly important words in terms of knowledge. A named entity refers to a proper noun (such as a person's name, organization name, or place name), date, time, quantity, amount, percentage, etc. For example, named entities may be extracted using a pre-trained Japanese expression extraction engine such as the Named Entity Extraction API (goo Labs), Apache OpenNLP, or CaboCha (LGPL / BSD license).

[0042] According to FIG. 3, the substantive word extraction unit 11 extracts substantive words from the above-mentioned utterance sentence and response sentence, for example, as follows. Entity of the utterance:book Richard Adams Response entity: Watership Down 1991 speculative fiction. Both the entity words of the utterance sentence and the entity words of the response sentence are output to the entity word pair extraction unit 12 .

[0043] [Entity word pair extraction part 12] The entity word pair extracting unit 12 extracts "entity word pairs" which are combinations of entity words. At this time, the entity word pair extracting unit 12 extracts pairs of entity words of the utterance sentence and entity words of the response sentence, and / or entity word pairs that are combinations of entity words of the response sentence.

[0044] According to FIG. 3, for example, the following entity word pairs are extracted from the entity words of the input utterance sentence and the entity words of the response sentence. (Richard Adams, Watership Down) (Richard Adams, 1991) (Richard Adams, speculative fiction) (book, Watership Down) (book, 1991) (book, speculative fiction) (Watership Down, 1991) (Watership Down, speculative fiction) (1991, speculative fiction)

[0045] According to Figure 3, (book, Richard Adams), which is contained only in the utterance, is deleted. In other words, there is no need to extract entity words in the utterance as entity word pairs. This is because the utterance is spoken by the user himself, and there is no need to detect knowledge errors. In this way, unnecessary entity word pairs are reduced as much as possible, and only necessary knowledge is narrowed down.

[0046] FIG. 4 is an explanatory diagram of the related word inference unit in the present invention.

[0047] Among the entity word pairs extracted by the entity word pair extraction unit 12, there are some that do not appear to have a direct relationship. However, it is difficult to automatically detect these as knowledge errors. For example, the entity word pair (1911, speculative fiction) does not appear to have a direct relationship, but it is difficult to determine that it "should not be taken into consideration when determining whether it is a knowledge error." For this reason, the relational word inference unit 13 infers the relational words in the entity word pairs, further narrowing down the entity word pairs.

[0048] [Related Word Inference Unit 13] The relative term inference unit 13 infers, from the utterance sentence and / or the response sentence, the relative terms in the entity word pairs extracted by the entity word pair extraction unit 12. This makes it possible to determine whether the relative terms are in an entity word pair that exists, and to obtain triples in an expression format in which the entity words are linked by the relative terms.

[0049] According to FIG. 4, the relational word inference unit 13 comprises a dialogue relational word inference engine and an entity word pair reduction function.

[0050] (Dialogue related word prediction engine) The dialogue relational word inference engine infers relational words in entity word pairs from utterances and / or response sentences. For example, OpenNRE can be used as a pre-trained language model that has undergone transfer learning (see, for example, Non-Patent Document 3). This allows the engine to extract what knowledge is contained in utterances and response sentences.

[0051] According to FIG. 4, the dialogue relational word inference engine infers the following triples, for example: (Richard Adams, (written-by), Watership Down) (Richard Adams, (wrote), speculative fiction) (book, (is a), Watership Down) (Watership Down, (release-year), 1991) (Watership Down, (has-genre), speculative fiction) These triples are output to the interaction graph generator 141 .

[0052] Here, even if the dialogue relative word estimation engine estimates the relative words in the entity word pairs, the likelihood (probability) of the estimated relative words may be low. In such cases, the engine may optionally have an entity word pair reduction function to delete entity word pairs with a likelihood equal to or lower than a predetermined value.

[0053] (Entity word pair reduction function) The entity word pair deletion function deletes entity word pairs extracted by the entity word pair extraction unit 12, the entity word pairs for which the relational words have not been estimated by the dialogue relational word estimation engine. According to FIG. 4, for example, for the following entity word pairs, the related words cannot be estimated with a high likelihood. (Richard Adams, 1991) (book, 1991) (book, speculative fiction) (1991, speculative fiction)

[0054] The entity word pair deletion function deletes entity word pairs for which related terms cannot be estimated with a high likelihood, and ultimately processes only the triples that are deemed necessary as knowledge. This prunes triples with low relevance, reducing the decline in judgment accuracy due to an increase in knowledge and the amount of memory consumed.

[0055] The reason for adopting the entity word pair removal function is that, for example, entity word pairs in which related words cannot be estimated with a high likelihood are assumed to be unable to be correct knowledge even if one of the entity words is replaced with another. In such cases, it is more likely that the related words have been forcibly predicted using unrelated entity word pairs, rather than due to an entity word error.

[0056] In another embodiment, the entity word pair deletion function may refer to the knowledge database 101. For example, when a triple of a related term (release-year) is predicted for the entity word pair (Richard Adams, 1911), the knowledge database 101 is referenced. Specifically, the function checks whether a triple such as (Richard Adams, release-year, "some entity word") exists, that is, whether knowledge exists in which the related term release-year is linked to the head entity Richard Adams. If such knowledge does not exist, the entity word pair can be deleted.

[0057] According to FIG. 4, the relational word inference unit 13 outputs the following entity word pairs to the knowledge graph generation unit 142. (Richard Adams, Watership Down) (Richard Adams, speculative fiction) (book, Watership Down) (Watership Down, 1991) (Watership Down, speculative fiction)

[0058] FIG. 5 is an explanatory diagram of the dialogue graph generation unit and knowledge graph generation unit in the present invention.

[0059] [Dialogue graph generation unit 141] The dialogue graph generating unit 141 generates a “dialogue graph” in which entity word pairs are connected by the relational words estimated by the relational word estimating unit 13 . A dialogue graph is generated by the entity word pairs and relational words output from the dialogue relational word inference engine of the relational word inference unit 13.

[0060] An "interaction graph" or "knowledge graph" is a graph that connects the triples of knowledge representation from the starting entity (head entity) to the ending entity (tail entity) using arrows to connect the relational terms (relations) that represent the relationships. As shown in Figure 5, for the triple (Watership Down, (release-year), 1991), an arrow connects the starting entity term (Watership Down) to the relative term (release-year) and then to the ending entity term (1991).

[0061] [Knowledge graph generation part 142] The knowledge graph generation unit 142 uses the knowledge database 101 to search for related terms in entity word pairs, and generates a "knowledge graph" that connects entity word pairs with the related terms. According to FIG. 5, for entity word pairs that have not been deleted by the entity word pair deletion function of the related word inference unit 13, related words are searched for using the knowledge database 101, and a knowledge graph is generated in which the entity word pairs are connected by the related words.

[0062] Each entity term in an entity term pair constitutes a node in the knowledge graph. An important point here is that even if a search of the knowledge database 101 fails to find a related term in the entity term pair and no triple exists, the node for that entity term remains. According to Figure 5, the entity word pair (Watership Down, 1991) is connected by a relational term (release-year) in the dialogue graph, but is not connected by a relational term in the knowledge graph. This means that although the triple (Watership Down, (release-year), 1991) is detected as knowledge in the response sentence to the user's utterance, this knowledge may be incorrect according to the knowledge database 101.

[0063] According to the conventional technology described above, because knowledge is processed as natural language, not only does the length of the input words increase, but it is also prone to problems such as a decrease in the accuracy of knowledge error detection and an increase in memory consumption. In contrast, according to the present invention, knowledge is processed in units of triples as graph data, which leads to improved accuracy in determining knowledge errors and reduced memory consumption.

[0064] FIG. 6 is an explanatory diagram of the feature generator and classification learning engine in the present invention.

[0065] [Feature Generation Unit 15] The feature generating unit 15 generates a "feature" (feature vector) for each entity word from the dialogue graph and the knowledge graph. The feature generator 15 is preferably a GNN (Graph Neural Network). For example, a CNN (Convolutional Neural Network) convolves information from eight directions, i.e., up, down, left, right, and diagonal directions, of an image. In contrast, a GNN convolves information about a node and other nodes linked to that node. This makes it possible to convert each node (entity word) of the graph into a high-dimensional vector to which a distributed representation generation algorithm (embedding) is applied.

[0066] According to Figure 6, the entity pair (Watership Down, 1991) is connected by a relational term (release-year) in the dialogue graph, but not by a relational term in the knowledge graph. In this case, there will be a difference between the feature values ​​in the dialogue graph and the feature values ​​in the knowledge graph for the entity terms (Watership Down) and (1991). In this case, it is difficult to determine which of the entity words in the triples (Watership Down) and (1991) is the knowledge error. However, the difference between the features obtained from the dialogue graph and the features obtained from the knowledge graph becomes large due to the relationship between the entity words and the relational words connected to other entity words. For example, for the entity word (Watership Down), suppose that the entity word (1991) is the incorrect knowledge and the correct entity word is (1990). In this case, according to Figure 6, the difference between the dialogue graph feature and the knowledge graph feature for the entity word (1991) is definitely larger than the difference between the dialogue graph feature and the knowledge graph feature for the entity word (Watership Down).

[0067] [Classification Learning Engine 16] In the dialogue graph and the knowledge graph, it is difficult to distinguish the feature quantities of nodes representing entity words simply by using a threshold distance. For this reason, the classification learning engine 16 is a pre-trained class classification classifier such as an FFN (Feed Forward Neural Network). The classification learning engine 16 is divided into an estimation stage and a training stage. In the training stage, a learning model is constructed from training data, as shown in Figure 7, which will be described later.

[0068] In the <estimation stage>, the classification learning engine 16 compares the feature amount of each entity word in the dialogue graph with the feature amount of each entity word in the knowledge graph, and classifies the knowledge as valid or incorrect. According to the present invention, knowledge errors in a response sentence to a user's utterance are detected, so the classification learning engine only needs to classify entity words contained in the response sentence. 6, the classification learning engine 16 determines that the entity word (1991) included in the response sentence is a knowledge error. This determination result is output to the dialogue graph display unit 17.

[0069] [Dialogue graph display section 17] The dialogue graph display unit 17 visually displays the dialogue graph, and also highlights the nodes corresponding to the entity words classified as knowledge errors. When the dialogue graph is displayed, the operator can see that the entity words (1991) classified as knowledge errors are highlighted and can also see the surrounding triple connections. This allows the operator to recognize the basis for determining the entity words that are knowledge errors in the response sentence.

[0070] <Classification learning engine training stage> FIG. 7 is a functional configuration diagram of a knowledge determination device in the training stage according to the present invention.

[0071] For the training stage, classification learning engine 16 in Fig. 7 has a dialogue database 102 instead of dialogue history storage unit 100 in Fig. 2. Furthermore, classification learning engine 16 in Fig. 7 further has a positive example graph generation unit 181 and a negative example graph generation unit 182. These functional components are also realized by executing a program that causes a computer installed in the device to function.

[0072] [Dialogue Database 102] The dialogue database 102 is a dialogue data set consisting of utterances and responses recorded from conversations between people based on correct knowledge. An example of the dialogue database 102 is OpenDialKG (see, for example, Non-Patent Document 4). This database specifically contains casual dialogues recommending books and movies. The system responders speak in a way that includes entity words based on correct knowledge. Note that, because this is a publicly available dataset that has been manually curated, related words with low degrees of relevance between entity words are excluded in order to reduce noise.

[0073] The utterances and responses in the dialogue database 102 are input to the entity word extraction unit 11. After that, the entity word extraction unit 11, entity word pair extraction unit 12, and relational word inference unit 13 are exactly the same as in the inference stage described above. The group of triples output from the relational word inference unit 13 is then output to the positive example graph generation unit 181.

[0074] FIG. 8 is an explanatory diagram of training the classification learning engine using the positive example graph generation unit and the negative example graph generation unit in the present invention.

[0075] [Positive Example Graph Generation Unit 181] The positive example graph generation unit 181 generates a positive example graph by connecting a predetermined range of entity word pairs extracted by the entity word extraction unit 11 and the entity word pair extraction unit 12 from the utterance sentences and response sentences in the dialogue database 102 with related words inferred by the related word inference unit 13.

[0076] [Negative example graph generation unit 182] The negative example graph generator 182 generates a negative example graph by replacing any entity word with another entity word from the positive example graph. The negative example graph is a graph in which an intentional knowledge error is caused in the positive example graph. Of course, the negative example graph may be generated by replacing or deleting related terms. Such entity word replacement may be performed automatically and randomly.

[0077] Thereafter, the feature generating unit 15 generates features for each entity word from the positive example graph and the negative example graph, respectively. This is also exactly the same processing as that of the feature generating unit 15 described above.

[0078] Then, in the training stage, the classification learning engine 16 inputs the following elements for each "entity word" and trains it. Entity word -> positive example graph features Negative graph features Label (positive / negative) The labels (positive and negative) are as follows: Knowledge validity: The entity word is not replaced from a positive example graph to a negative example graph. Knowledge error: The entity word is substituted from the substituted positive example graph to the negative example graph.

[0079] According to the positive example graph in Figure 8, if the entity word pair (Watership Down, 1990) is connected by a relational word (release-year), for example, a negative example graph is created in which the entity word (1990) is replaced with the entity word (1992).

[0080] Thereafter, the feature generator 15 generates features for each entity word from the positive example graph and the negative example graph, which is also similar to the GNN described above.

[0081] Then, in the training stage, the classification learning engine 16 performs training by inputting, for each entity word, the feature amounts of the positive example graphs, the feature amounts of the negative example graphs, and the labels (positive and negative) in association with each other. Here, a positive (knowledge correct) label is input for an entity word that has not been replaced as a negative example graph, and a negative (knowledge incorrect) label is assigned for an entity word that has been replaced as a negative example graph. Note that even for an entity word that has not been replaced, there will be a difference between the feature amount of the positive example graph and the feature amount of the negative example graph. This is because the configuration of the entire directed graph will be different when at least one entity word is replaced. The classification learning engine 16 is trained to input two features for each entity word and output a label (positive or negative) that will be used for class classification. The error is backpropagated, and the weights inside the feature generator 15 and the classification learning engine 16 are repeatedly updated.

[0082] As described above in detail, the knowledge determination device, program, and method of the present invention can detect knowledge errors in words in a sentence and clarify the basis for the determination without requiring annotation of related knowledge.

[0083] The present invention has the following advantages over the prior art. (A) By visually checking the dialogue graph among the connections of knowledge contained in the utterance sentence and the response sentence, the basis for determining whether there is a knowledge error becomes clear. (B) It does not require prior manual annotation of related knowledge (e.g., related terms), and automatically extracts related knowledge from existing knowledge databases, so it does not affect the accuracy of knowledge error detection. (C) It is possible to detect knowledge errors in words contained in utterances and responses. (D) It is based on the dialogue structure of utterances and responses, and does not extract unnecessary connections of knowledge.

[0084] Furthermore, this will enable, for example, "a dialogue system free of knowledge errors," which will contribute to Goal 8 of the United Nations' Sustainable Development Goals (SDGs), "Promote inclusive and sustainable economic growth, employment and decent work for all."

[0085] With respect to the various embodiments of the present invention described above, various changes, modifications, and omissions that fall within the scope of the technical spirit and aspects of the present invention may be easily made by those skilled in the art. The above description is merely illustrative and is not intended to be limiting in any way. The present invention is limited only by the claims and their equivalents. [Explanation of symbols]

[0086] 1 Knowledge judgment device 100 Dialogue history storage unit 101 Knowledge Database 102 Interactive Database 11 Entity Extraction 12 Entity pair extraction unit 13 Related Words Prediction Unit 141 Interaction graph generation unit 142 Knowledge graph generation unit 15 Feature generation unit 16 Classification Learning Engine 17 Interactive graph display section 181 Positive Example Graph Generation Unit 182 Negative Example Graph Generation Unit 2. Devices, smartphones 3. Conversation Server

Claims

1. In a knowledge determination device for detecting knowledge errors in a sentence, A knowledge database that stores knowledge that links entity words with relational words, a substance word extraction means for extracting substance words that satisfy a predetermined condition from a sentence; an entity word pair extraction means for extracting entity word pairs each of which is a combination of entity words; a relative word inference means for inferring a relative word in the entity word pair from the sentence; a dialogue graph generating means for generating a dialogue graph in which entity word pairs are connected by the relational words estimated by the relational word estimation means; a knowledge graph generating means for searching for related terms in entity word pairs using a knowledge database and generating a knowledge graph connecting entity word pairs with the related terms; a feature generating means for generating a feature for each entity word from each of the dialogue graph and the knowledge graph; a classification learning engine that inputs, for each entity word, the feature value of the entity word in the dialogue graph and the feature value of the entity word in the knowledge graph and classifies the knowledge as correct or incorrect; A knowledge determination device comprising:

2. To detect knowledge errors in responses to user utterances, The entity word extraction means extracts entity words that satisfy a predetermined condition from the utterance sentence and the response sentence; the entity word pair extraction means extracts pairs of entity words of the utterance sentence and entity words of the response sentence, and / or entity word pairs that are combinations of entity words of the response sentence; The classification learning engine classifies the entity words contained in the response sentence.

2. The knowledge determination device according to claim 1.

3. The entity word extraction means extracts entity words of named entities and / or pre-registered entity words as a predetermined condition.

3. The knowledge determination device according to claim 1 or 2.

4. A dialogue graph display means is provided for visually displaying the dialogue graph and highlighting the nodes of entity words classified as knowledge errors.

3. The knowledge determination device according to claim 1, further comprising:

5. The related word estimation means is a dialogue relational word inference engine for inferring relational words in entity word pairs from the sentence; an entity word pair deletion function for deleting entity word pairs for which related words have not been estimated by the dialogue related word estimation engine from among the entity word pairs extracted by the entity word pair extraction means; 3. The knowledge determination device according to claim 1, further comprising:

6. The dialogue graph generating means generates a dialogue graph in which entity word pairs are connected by the relational words estimated by the dialogue relational word estimation engine of the relational word estimation means, The knowledge graph generating means uses the knowledge database to search for related terms for entity word pairs that have not been deleted by the entity word pair deletion function of the related term inference means, and generates a knowledge graph that connects entity word pairs with the related terms.

6. The knowledge determination device according to claim 5.

7. The knowledge database is a collection of correct knowledge expressed in triples, which link entity words with relational terms.

3. The knowledge determination device according to claim 1 or 2.

8. The feature generation means is a GNN (Graph Neural Network), The classification learning engine is a Feed Forward Neural network (FFN).

3. The knowledge determination device according to claim 1 or 2.

9. To pre-train the classification learning engine, Instead of the sentences, a dialogue database is provided that stores utterances and responses based on correct knowledge, a positive example graph generating means for generating a positive example graph by connecting a predetermined range of entity word pairs extracted by the entity word extraction means and the entity word pair extraction means from the utterance sentences and the response sentences of the dialogue database with the related word inferred by the related word inferring means; a negative example graph generating means for generating a negative example graph by replacing any entity word with another entity word and / or replacing any related word with another related word or deleting the related word from the positive example graph; and The feature generating means generates a feature for each entity word from each of the positive example graph and the negative example graph, In the training stage, the classification learning engine is trained to input the feature quantities of the positive example graph and the feature quantities of the negative example graph for each entity word, and to output a label indicating correct knowledge if the entity word is not substituted as the negative example graph, and to output a label indicating incorrect knowledge if the entity word is substituted.

3. The knowledge determination device according to claim 1 or 2.

10. A program for causing a computer installed in a device for detecting knowledge errors in a sentence to function, A knowledge database that stores knowledge that links entity words with relational words, a substance word extraction means for extracting substance words that satisfy a predetermined condition from a sentence; an entity word pair extraction means for extracting entity word pairs each of which is a combination of entity words; a relative word inference means for inferring a relative word in the entity word pair from the sentence; a dialogue graph generating means for generating a dialogue graph in which entity word pairs are connected by the relational words estimated by the relational word estimation means; a knowledge graph generating means for searching for related terms in entity word pairs using a knowledge database and generating a knowledge graph connecting entity word pairs with the related terms; a feature generating means for generating a feature for each entity word from each of the dialogue graph and the knowledge graph; a classification learning engine that inputs, for each entity word, the feature value of the entity word in the dialogue graph and the feature value of the entity word in the knowledge graph and classifies the knowledge as correct or incorrect; A program that causes a computer to function.

11. A knowledge determination method for a device for detecting knowledge errors in a sentence, comprising: The device has a knowledge database that stores knowledge that links entity words with relational words, A first step of extracting entity words that satisfy a predetermined condition from a sentence; a second step of extracting entity word pairs, which are combinations of entity words; a third step of estimating a related word in the entity word pair from the sentence; a fourth step of generating a dialogue graph connecting the entity word pairs with the relational terms estimated in the third step, and searching for the relational terms in the entity word pairs using a knowledge database to generate a knowledge graph connecting the entity word pairs with the relational terms; a fifth step of generating features for each entity word from each of the dialogue graph and the knowledge graph; a sixth step of inputting, for each entity word, the feature quantity of the entity word in the dialogue graph and the feature quantity of the entity word in the knowledge graph using a classification learning engine, and classifying the knowledge into correct knowledge or incorrect knowledge; A knowledge determination method characterized by executing the above.

Images