Concept prediction for creating new intents and automatically assigning examples in dialogue systems

Abstract

A method for automatically creating new intents and assigning examples to the new intents in a dialogue system includes generating at least one predicted intent from an intent concept graph by combining two or more nodes of the intent concept graph. A graph embedding generates the at least one predicted intent. A sentence embedding is generated from examples from a call recording. The graph embedding of the at least one predicted intent is then matched with examples from the sentence embedding, and the at least one predicted intent and examples are connected to the dialogue system.

Description

[Technical Field]

[0001] The present disclosure relates generally to conversational artificial intelligence computer systems, and more particularly to a concept prediction method and system for creating new intents and automatically assigning examples in a dialogue system. [Background technology]

[0002] The intent-action paradigm used today by computerized conversational systems like the IBM Watson® Assistant (WA) framework requires chatbot developers to provide an extensive list of each intent used in the system. Medium-sized and large dialogue systems can contain thousands of intents. Call recordings can contain intent examples that have not yet been processed by the dialogue system. Such examples can represent either existing intents or new intents that may be added to the system. This process currently involves domain experts (DEs) curating examples by manually creating new intents and assigning these examples to the new intents.

[0003] Chatbot developers / domain experts typically manually match sentences in call recordings to new intents or intents from a large collection of existing intents. Finding examples from chat logs or human call recordings is unsuccessful, time-consuming, and error-prone.

[0004] A domain expert can offline define intents for possible user utterances through a list of examples (e.g., >10) and ask the system for recommendations, then define entities. These examples are processed by the conversational system, which learns a mapping from user utterances to intents. Summary of the Invention

[0005] According to various embodiments, a computing device, a non-transitory computer-readable storage medium, and a method are provided for automatically creating new intents and assigning examples to those new intents in a dialogue system.

[0006] In one embodiment, a computer-implemented method includes generating at least one predicted intent from the intent concept graph by combining two or more nodes of the intent concept graph. A graph embedding of the at least one predicted intent is generated. A sentence embedding is generated from examples from the call recording. Then, the at least one predicted intent from the graph embedding is matched with examples from the sentence embedding, and the at least one predicted intent and examples are connected to a dialogue system. By representing intents in the intent concept graph, the method can automatically generate new intents without requiring the time and effort of manual user input.

[0007] In some embodiments, the at least one predicted intent includes a plurality of predicted intents, and the method further includes ranking the plurality of predicted intents. In some embodiments, the ranking of the plurality of predicted intents is based on the number of nodes that two or more nodes of the predicted intents have in common. In some embodiments, the edges between connected nodes of the intent concept graph are directional edges.

[0008] In some embodiments, the method further includes verifying the at least one predicted intent in terms of performance against examples from the call log. In some embodiments, verifying the at least one predicted intent in terms of performance is performed automatically. In some embodiments, verifying the at least one predicted intent in terms of performance is performed by displaying the at least one predicted intent and examples that match it for user verification.

[0009] In some embodiments, the method further includes performing a random walk of the intent concept graph to generate at least one predicted intent.

[0010] In some embodiments, the method further includes matching language from the call recording to existing nodes of the intent concept graph to generate at least one predicted intent of the intent concept graph.

[0011] According to various embodiments, a computerized conversation system includes a computerized conversation agent configured to receive a query from an end user and output an answer to the end user, and a concept prediction module configured to automatically predict at least one new intent based on a set of existing intents represented by an intent concept graph and automatically associate the at least one new intent with at least one example in a call recording.

[0012] According to various embodiments, a non-transitory computer-readable storage medium tangibly embodies computer-readable program code having computer-readable instructions that, when executed, cause a computing device to perform a method for improving the computing efficiency of a computerized dialogue system. The method includes generating at least one predicted intent from an intent concept graph by combining two or more nodes of the intent concept graph. The method further includes generating a graph embedding of the at least one predicted intent and generating sentence embeddings from examples from the call recording. The method further includes matching the at least one predicted intent with examples from the sentence embeddings and connecting the at least one predicted intent and examples to the dialogue system.

[0013] The concepts discussed herein provide tools for improving knowledge transfer to computerized conversational systems. These concepts ensure the expressiveness, scalability, and efficiency of such conversational systems, and can further enable the discovery of conflicts and interferences between sets of intents, while supporting the search for knowledge domains not covered by existing intents.

[0014] These and other features will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.

[0015] The drawings are of exemplary embodiments. They do not illustrate all embodiments. Other embodiments may be used in addition or instead. To save space or for a more effective explanation, details that may be obvious or unnecessary may be omitted. Some embodiments may be practiced with additional components or steps, or without all of the components or steps shown, or combinations thereof. When the same numeral appears in different drawings, it refers to the same or similar components or steps. [Brief explanation of the drawings]

[0016] [Figure 1A] 1 is a flowchart illustrating an overall method for automatically creating new intents and automatically assigning examples with the new intents in a dialogue system, consistent with an illustrative embodiment. [Figure 1B] 1 is a flowchart illustrating an overall method for automatically creating new intents and automatically assigning examples with the new intents in a dialogue system, consistent with an illustrative embodiment. [Figure 2] 1 is a graphical representation illustrating steps followed by a system for automatically creating new intents and automatically assigning examples with the new intents, consistent with an illustrative embodiment; [Figure 3]1 is an exemplary word graph illustrating initial training for adding knowledge to a word graph. [Figure 4] 1 is an example conceptual graph illustrating automatic prediction of new nodes in a conceptual graph. [Figure 5] FIG. 1 is a functional block diagram illustration of a computer hardware platform that can be used to implement a specially configured computing device capable of hosting a system for automatically creating new intents and automatically assigning examples with the new intents in a dialogue system, consistent with an illustrative embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] In the following detailed description, numerous specific details are set forth by way of example in order to provide a thorough understanding of the relevant teachings. However, it should be apparent that the present teachings may be practiced without such details. In other instances, well-known methods, procedures, components, or circuits, or combinations thereof, have been described at a relatively high-level, without detail, so as not to unnecessarily obscure aspects of the present teachings.

[0018] Unless otherwise indicated, and as will be apparent from the description and claims that follow, descriptions using terms such as "processing," "computing," "calculating," "determining," and the like throughout this specification should be understood to refer to the operation or processing of a computer or computing system, or combination thereof, or similar electronic computing device that manipulates and / or transforms data represented as electronic physical quantities in the computing system's registers or memory into other data similarly represented as physical quantities in the computing system's memory, registers, or other information storage, transmission, or display device.

[0019] As described in more detail below, the present disclosure generally relates to a system and computerized method for automatically creating new intents and automatically assigning examples with the new intents in a dialogue system, also referred to herein as a computerized dialogue system or computerized conversation system. The system and computerized method provide technical improvements in the efficiency and accuracy of computerized systems by speeding the creation, deployment, and integration of concepts identified in call records and mapped to intents in a computerized conversation system. By automatically creating new intents and automatically assigning examples with the new intents, the system and computerized method can ensure expressiveness, scalability, and efficiency in a computerized conversation system.

[0020] As used herein, a "computerized conversation system," or "virtual assistant," or "dialog system," refers to an intent-based conversational artificial intelligence system that can derive an intent from an end-user's query and provide a predetermined answer based on that intent. An "intent" refers to the end-user's motivation. It refers to the intention behind each message received by a computerized conversation system. An intent relates to what the user wants to get out of the interaction. For example, if an end-user asks, "How much is a late payment?" the system can determine that the intent behind the message is to verify the user's account information and provide the amount of the late fee on that account. Thus, an intent-based computerized conversation system can operate to identify the meaning that a message from an end-user is intended to convey, rather than relying on specific input.

[0021] Intent-based computerized conversational systems look for important terms or "entities" in received messages. Entities are variables that inform the intent of the message. For example, a user can ask the system, "What's the weather like in New York on Friday?" The intent is to find out the weather. The terms "New York" and "Friday" are entities that identify the information the user wants.

[0022] Call recordings can contain intent examples that have not yet been processed by the dialogue system. Such examples can represent either existing intents or new intents that may be added to the system. As discussed in more detail below, aspects of the present disclosure can take advantage of the representation of intents as graphs, where each intent is represented by a set of interconnected concepts, which can be either high-level concepts extracted from examples or low-level word clouds. The concept graph can be used to predict new intents and associate them with call recording examples. The predicted new intents can be visualized in a graphical user interface by a domain expert, who can manually verify the system's results if necessary.

[0023] Reference is now made to the details illustrated in the accompanying drawings and described below.

[0024] 1A and 1B, a method 100 for automatically creating new intents and automatically assigning examples with new intents in a dialogue system is shown consistent with an exemplary embodiment. In operation 105, input is provided, where a graph is provided having intents represented as interconnected concepts and a word cloud, which are words associated with a low-level representation of the concepts. Intents are either computed from intent examples or manually associated with concepts by a user. An example of such an intent-concept graph is described below with respect to FIG. 3.

[0025] The intent concept graph can be used in operation 110 to suggest new, probable intents using a score based on the number of nodes two nodes have in common. This differs from traditional link prediction because the intent concept graph has a direction associated with it, and accordingly, traditional link prediction does not work. The directional edges, as described below, indicate the general order of words placed in the predicted concept. A random walk through the intent concept graph can also be used, as new intents may be different walkthroughs of different intents. Additionally, examples from call recordings 111 may be used to help determine new intents.

[0026] Once a new intent is identified, the new intent may be ranked based on the intent prediction score in operation 120. Given that an intent may be represented by a sequence of concepts, the new intent may be positioned at the beginning, middle, or end of the sequence of concepts.

[0027] A word cloud for the new intent may be generated at operation 130. A graph embedding for the new intent may be generated at operation 140. This may be performed using standard graph embedding methods such as DeepWalk, NodeToVec, or graph convolutional neural networks, for example.

[0028] In operation 150, a sentence embedding for each example call record may be generated. This may be performed, for example, by sentence embedding methods such as BERT and USE.

[0029] At operation 160, the new intent embeddings can be matched with example embeddings from call recordings 111, for example, by utilizing standard machine learning mapping methods.

[0030] At operation 170, the new intent can be connected with examples in the dialogue system as a result of the mapping performed at operation 160. At operation 180, the inclusion of the new proposed intent is validated from a performance perspective, which can be done in an automated manner without requiring input from a user or domain expert. For example, the new proposed intent can be used in a call record response to determine whether an appropriate response is generated. However, the user can be involved in this process to consider whether these new intents and examples are satisfactory. In this regard, a user interface can be provided that allows the user to review the new intents and examples as a further validation.

[0031] If the new proposed intent is approved for inclusion (accept=YES), then in operation 190 the intent concept graph and word cloud from the input are updated, as shown in operation 195. If the new proposal is not approved for inclusion (accept=NO), the new intent is discarded and the graph from the input remains the same.

[0032] FIG. 2 is a schematic diagram illustrating an example of data flow 200 in method 100 of FIGS. 1A and 1B. Additional details are provided below with respect to FIGS. 3 and 4. A concept graph 202 can include linking words from examples to intents. The system can form new intent predictions 204 by linking concepts or by a random walk on the concept graph. The system can embed new intents into the concept graph by graph embedding 206, such as deepwalk, NodeToVec, etc. Additionally, call record examples 208 can be used for sentence embedding 210. The graph embedding 206 and sentence embedding 210 can be used for mappings 212 learned from intent examples, for example, by a mean squared error (MSE) method.

[0033] Figure 3 illustrates an example word graph and taxonomy 300. Nodes with darker backgrounds, such as the node for chatbot 302, are concept nodes. Nodes with lighter backgrounds, such as the node for name 304, are word nodes. Nodes may be connected by edges 306 that connect to form classes 308. The chart below shows example classes with training examples, such as examples from call recordings, used to specify the classes. TIFF0007737208000001.tif77169

[0034] TIFF0007737208000002.tif62167

[0035] TIFF0007737208000003.tif102167

[0036] Referring to Figure 4, a conceptual graph 400 is shown with link predictions. Here, solid edges, such as edge 402, indicate edges already in the conceptual graph. Dashed edges, such as dashed edge 404, indicate intents 408 that may be formed from linked concepts (or notes 406) in conceptual graph 400. Based on the above table of classes from training examples, a further class of "get_account_insurance" may be an intent prediction. Examples of this intent may be obtained from call recordings, as described above, to provide an example dialogue system that can be mapped to a predicted intent.

[0037] The new intent clustering algorithm may involve finding new intents with link predictions on a taxonomy graph like the graph in Figure 3, extending the word graph and taxonomy with the word graph of the predicted new intents, recomputing graph embedding and MSE mapping, mapping utterances from chat transcripts to the extended intent space, looking for predicted intents with a high number of utterances nearby, and returning the top predicted intents with associated taxonomies and examples. The number of nodes common to words or concepts may be used to rank the predicted intents.

[0038] While the "get_account_insurance" example above is a prediction of a new node, the system can also predict new edges. In the example of Figure 4, a new edge may be established between "name" and "account." With such a new edge, the system may predict a new intent, for example, "get_account_name."

[0039] 5 provides an example functional block diagram of a computer hardware platform 500 that can be used to implement a particularly configured computing device capable of hosting a concept prediction system 520 for a computerized conversation system, such as the concept prediction system described above with respect to method 100. The concept prediction system 520 can predict concepts in order to automatically create new intents and assign examples to those intents. In particular, FIG. 5 illustrates a network or host computer platform 500 such as may be used to implement a suitably configured concept prediction system 520.

[0040] The computer platform 500 may include a central processing unit (CPU) 504, a hard disk drive (HDD) 506, a random access memory (RAM) or read-only memory (ROM) 508 or a combination thereof, a keyboard 510, a mouse 512, a display 514, and a communication interface 516 connected to a system bus 502.

[0041] In one embodiment, HDD 506 has functionality that includes storing programs capable of performing various processes, such as concept prediction system 520, in the manner described herein. Concept prediction system 520 may have various modules configured to perform different functions. For example, it may include concept graph 202, sentence embedding module 210, graph embedding module 206, and example call record 208, as described above with respect to FIG. 2.

[0042] In one embodiment, it may store programs such as Apache™ for operating the system as a web server. In one embodiment, HDD 506 may store running applications, including one or more library software modules, such as those for the Java™ Runtime Environment program for implementing a JVM (Java™ Virtual Machine).

[0043] <Conclusion> The description of various embodiments of the present teachings has been presented for purposes of illustration and is not intended to be exhaustive or limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used herein have been selected to best explain the principles of the embodiments, practical applications or technical improvements over technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0044] While the foregoing describes what is considered to be the best mode and / or alternative embodiments, it is understood that various modifications may be made thereto and that the subject matter disclosed herein can be embodied in various forms and examples, and that the present teachings are susceptible to numerous applications, only some of which are described herein. It is intended by the following claims to claim all such applications, modifications, and variations that fall within the true scope of the present teachings.

[0045] The components, steps, features, objects, benefits, and advantages discussed herein are merely exemplary. None of them, nor the discussions related thereto, are intended to limit the scope of protection. While various advantages have been discussed herein, it will be understood that not all embodiments necessarily include all advantages. Unless otherwise specified, all measurements, values, ratings, positions, sizes, and other specifications set forth in this specification, including the following claims, are approximate rather than precise. They are intended to have a reasonable range consistent with the functions to which they pertain and with those customary in the technical field to which they pertain.

[0046] Numerous other embodiments are contemplated, including embodiments having fewer, additional, or different components or combinations thereof, steps, features, objects, benefits, and advantages, including embodiments in which the components and / or steps are arranged and / or ordered differently.

[0047] Embodiments of the present invention are described herein with reference to flowchart and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. Each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer-readable program instructions.

[0048] The computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, whereby the instructions, executed by the processor of such computer or other programmable data processing apparatus, create means for performing the functions / acts identified in one or more blocks of the flowcharts and / or block diagrams. The computer-readable program instructions may also be stored on a computer-readable storage medium capable of instructing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner. The computer-readable storage medium having instructions stored thereon thereby constitutes an article of manufacture including instructions for performing aspects of the functions / acts identified in one or more blocks of the flowcharts and / or block diagrams.

[0049] Computer-readable program instructions may also be loaded into a computer, other programmable device, or other device and a series of operational steps executed on the computer, other programmable device, or other device to create a computer-implemented process, whereby the instructions executing on the computer, other programmable device, or other device perform the functions / operations identified in one or more blocks in the flowcharts and / or block diagrams.

[0050] The call flows, flowcharts, and block diagrams in the figures of this disclosure illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for performing a specific logical function. In some alternative implementations, the functions shown in the blocks may be executed in a different order than shown in the figures. For example, two blocks shown in succession may actually be executed concurrently or nearly concurrently, or may even be executed in reverse order, depending on the functionality involved. Note that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs a specific function or operation or executes a combination of dedicated hardware and computer instructions.

[0051] While the foregoing has been described in connection with exemplary embodiments, it is understood that the term "exemplary" is meant merely as an example, not as best or optimal. Except as noted immediately above, nothing described or illustrated is intended to, and should be construed as, dedicating to the public any component, step, feature, object, benefit, advantage, or equivalent, whether claimed or not.

[0052] Terms and expressions used herein will be understood to have the ordinary meanings given to such terms and expressions with respect to their respective corresponding fields of study and research, unless a specific meaning is indicated herein. Relationship terms, such as first and second, are used solely to distinguish one entity or act from another and do not necessarily require or imply an actual relationship or sequence between such entities or acts. The terms "comprise," "comprising," or other variations are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus consisting of a list of elements does not include only those elements, but may include other elements not expressly listed or elements inherent in such process, method, article, or apparatus. The use of an element preceded by "a" or "an" does not, without further constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0053] The Abstract of the Disclosure is provided to enable the reader to quickly grasp the contents of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Moreover, in the foregoing Detailed Description, it will be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments have more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Accordingly, the following claims are hereby incorporated into this specification, with each claim standing on its own as separately claimed subject matter.

Claims

1. generating at least one predicted intent from an intent concept graph including a plurality of nodes by combining two or more nodes among the plurality of nodes; generating a graph embedding of the at least one predicted intent; and generating sentence embeddings from examples from call recordings; matching the at least one predicted intent in the graph embedding with the examples from the sentence embedding; and connecting the at least one predicted intent with the example in a dialogue system.

2. The computer-implemented method of claim 1 , wherein the at least one predicted intent comprises a plurality of predicted intents, the method further comprising ranking the plurality of predicted intents.

3. The computer-implemented method of claim 2 , further comprising ranking the plurality of predicted intents based on a number of nodes that two or more nodes of the predicted intents have in common.

4. The computer-implemented method of claim 3 , wherein edges between connected nodes of the intent conceptual graph are directional edges.

5. The computer-implemented method of claim 1 , further comprising validating the at least one predicted intent in terms of performance against examples from the call recording.

6. The computer-implemented method of claim 5 , further comprising automatically validating the at least one predicted intent in terms of performance against examples from the call recording.

7. Validating the at least one predicted intent from a performance perspective includes: displaying the at least one predicted intent and the matching examples on a user interface of a computing device; and and receiving a confirmation from the user via the user interface.

8. The computer-implemented method of claim 1 , further comprising performing a random walk of the intent concept graph to generate the at least one predicted intent.

9. 2. The computer-implemented method of claim 1, further comprising matching language from the call recording to existing nodes in the intent concept graph to generate the at least one predicted intent in the intent concept graph.

10. A computerized conversational agent configured to receive queries from an end user and output answers to said end user; a concept prediction module configured to automatically predict at least one new intent based on a set of existing intents represented by an intent concept graph, and automatically associate the at least one new intent with at least one example in a call recording; The concept prediction module: generating graph embeddings for the at least one new intent and sentence embeddings from examples from the call recording; matching the at least one new intent from the graph embedding with the examples from the sentence embedding; and connecting said at least one new intent with said examples in a dialogue system.

11. 11. The computerized conversation system of claim 10, wherein the at least one new intent includes a plurality of new intents, and the concept prediction module is further configured to rank the plurality of new intents based on a number of nodes that two or more nodes of the predicted intents have in common.

12. 11. The computerized conversation system of claim 10, wherein the concept prediction module is configured to automatically validate the at least one new intent in terms of performance against examples from the call recording.

13. 11. The computerized conversation system of claim 10, wherein the concept prediction module is configured to display the at least one new intent and the associated examples in a user interface and receive confirmation from a user via the user interface.

14. A non-transitory computer-readable storage medium tangibly embodied with computer-readable program code having computer-readable instructions that, when executed, cause a computing device to perform a method for improving the computing efficiency of a computing device operating an interactive system, the method comprising: generating at least one predicted intent from the intent concept graph by combining two or more nodes of the intent concept graph; generating a graph embedding of the at least one predicted intent; and generating sentence embeddings from examples from call recordings; matching the at least one predicted intent from the graph embedding with the examples from the sentence embedding; and connecting the at least one predicted intent with the examples in a dialogue system.

15. the at least one predicted intent includes a plurality of predicted intents; 15. The non-transitory computer-readable storage medium of claim 14, wherein the computer-readable instructions, when executed, further configure the computing device to perform operations including ranking the plurality of predicted intents based on a number of nodes that two or more nodes of the predicted intents have in common.

16. The non-transitory computer-readable storage medium of claim 14, wherein the computer-readable instructions, when executed, further configure the computing device to perform operations including automatically validating the at least one predicted intent in terms of performance against examples from the call logs.

17. The computer-readable instructions, when executed, displaying the at least one predicted intent and the matching examples on a user interface of the computing device; and and receiving a confirmation from a user via the user interface.

18. The non-transitory computer-readable storage medium of claim 14, wherein the computer-readable instructions, when executed, further configure the computing device to perform operations including performing a random walk of the intent concept graph to generate the at least one predicted intent.

19. The non-transitory computer-readable storage medium of claim 14, wherein the computer-readable instructions, when executed, further configure the computing device to perform operations including matching language from the call records to existing nodes in the intent conceptual graph to generate the at least one predicted intent in the intent conceptual graph.

Images