System, method, and program for improving the performance of a dialogue system using a dialogue agent

Abstract

A system, program, and method are provided for improving the performance of a dialogue system using an automated virtual dialogue agent, including utilizing the automated virtual agent to receive a natural language request and generate a corresponding response, automatically identifying and resolving a corresponding knowledge gap between the request and the response, and using the resolved knowledge gap to refine the automated virtual agent.

Description

[Technical Field]

[0001] The present embodiments relate to virtual dialogue systems that employ automated virtual dialogue agents, such as "chatbots," as well as related computer program products and computer-implemented methods. In certain example embodiments, to improve the performance of the automated virtual dialogue agents, knowledge gaps between one or more requests and corresponding expected responses are identified and resolved using solutions aimed at bridging or minimizing the knowledge gaps. [Background technology]

[0002] A chatbot is a computer program that uses artificial intelligence (AI) as a platform for conducting transactions between automated virtual conversational agents and users, typically consumers. Transactions may involve product sales, customer service, information retrieval, or other types of transactions. Chatbots often interact with users through written (e.g., online or text) or auditory (e.g., telephone) interactions. It is known in the art that chatbots function as a question-answering component between users and AI platforms. The quality of questions and answers is derived from the quality of question understanding, question translation, and answer resolution. A common cause of failure to find a response corresponding to a question is the lack of knowledge regarding the effective translation of the question into an equivalent knowledge representation that maps to the answer. For example, the lack of synonyms or conceptual relationships can limit the AI ​​platform's ability to determine whether a question is equivalent to or related to a known question for which an answer is available. Summary of the Invention

[0003] Embodiments include systems, computer program products, and methods for improving the performance of dialogue systems, and in certain embodiments, the improvements are directed to active explanations to dynamically solicit input regarding relevant knowledge to bridge knowledge gaps.

[0004] In one aspect, a system is provided for use with a computer system including a processing unit (e.g., a processor) operably coupled to a memory and an artificial intelligence (AI) platform in communication with the processing unit. The AI ​​platform is configured with tools for managing the performance of the operably coupled dialogue system. These tools include a dialogue manager, an artificial intelligence (AI) manager, and a director. The dialogue manager functions to receive and process natural language (NL) associated with interactions with an automated virtual dialogue agent of the dialogue system. The NL includes dialogue events in the form of one or more input instances and one or more corresponding output instances or output behaviors. The AI ​​manager functions to apply the dialogue events to a learning program to interpret one or more requests, identify knowledge gaps, and dynamically bridge the knowledge gaps. The director functions to refine the automated virtual dialogue agent in response to the knowledge gaps being bridged, with refinements aimed at improving the performance of the dialogue system.

[0005] In another aspect, a computer program product is provided for improving the performance of a virtual dialogue agent system. The computer program product includes a computer-readable storage medium having program code embodied thereon. The program code is executable by a processor to manage the performance of an operably coupled dialogue system. The program code is operative to receive and process natural language (NL) associated with the dialogue system's interaction with an automated virtual dialogue agent. The NL includes dialogue events in the form of one or more input instances and one or more corresponding output instances or output actions. The program code is further operative to apply the dialogue events to a learning program to interpret one or more requests, identify knowledge gaps, and dynamically bridge the knowledge gaps. Program code is further provided for improving the automated virtual dialogue agent in response to the knowledge gaps being bridged, with the improvements intended to improve the performance of the dialogue system.

[0006] In yet another aspect, a computer-implemented method for improving performance of a dialogue system is provided. The method includes receiving and processing, by a processor of a computing device, natural language (NL) associated with a dialogue with an automated virtual dialogue agent. The NL includes dialogue events in the form of one or more input instances and one or more corresponding output instances or output actions. The dialogue events are applied to a learning program to interpret the one or more input instances, identify knowledge gaps, and dynamically bridge the knowledge gaps. The automated virtual dialogue agent is refined according to the knowledge gaps bridged, with the refinement intended to improve performance of the dialogue system.

[0007] In a further aspect, a computer system includes an artificial intelligence (AI) platform in communication with a processor. The AI ​​platform is configured with tools for managing the performance of an operably coupled dialogue system. These tools include a dialogue manager, an artificial intelligence (AI) manager, and a director. The dialogue manager functions to receive and process natural language (NL) associated with a dialogue with an automated virtual dialogue agent of the dialogue system. The AI ​​manager functions to apply dialogue events to a learning program to interpret one or more input instances, identify knowledge gaps, and dynamically bridge the knowledge gaps. The director functions to refine the automated virtual dialogue agent in response to the knowledge gaps being bridged, with refinements aimed at improving the performance of the dialogue system.

[0008] These and other features and advantages will become apparent from the following detailed description of the exemplary embodiments taken in conjunction with the accompanying drawings.

[0009] The drawings referenced herein form a part of this specification and are incorporated herein by reference. Features shown in the drawings are intended to be illustrative of only some embodiments and not all embodiments, unless expressly stated otherwise. [Brief explanation of the drawings]

[0010] [Figure 1] This is a system diagram showing the computing system of an artificial intelligence platform in a network environment. [Figure 2] FIG. 2 is a block diagram illustrating the tools of the artificial intelligence platform shown and described in FIG. 1 and their associated application program interfaces. [Figure 3] 1 is a flow chart illustrating an embodiment of a method for enriching a corpus of reliable domain-specific semantic relations. [Figure 4] 1 is a flow chart illustrating an embodiment of a method for knowledge-enhanced interaction. [Figure 5] 1 is a flow chart illustrating an embodiment of a method for generating an explanation. [Figure 6] 1 is a flow chart illustrating an embodiment of a method for generating explanation alternatives as semantic relationships between question and answer phrases to enhance domain knowledge. [Figure 7] 10 is a flowchart illustrating processing and recording descriptions of relationships between questions and answers. [Figure 8] FIG. 2 is a block diagram illustrating an exemplary relation-knowledge artifact-map. [Figure 9] FIG. 10 is a block diagram illustrating an example computer system / server of a cloud-based support system for implementing the systems and processes described above with respect to FIGS. 1-8. [Figure 10] FIG. 1 is a block diagram illustrating a cloud computing environment. [Figure 11] FIG. 1 is a block diagram illustrating a series of functional abstraction model layers provided by a cloud computing environment. DETAILED DESCRIPTION OF THE INVENTION

[0011] It will be readily understood that the components of the present embodiments, as generally described and illustrated in the Figures herein, could be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the present apparatus, system, method, and computer program product embodiments as illustrated in the Figures is not intended to limit the scope of the claimed embodiments, but is merely representative of selected embodiments.

[0012] Throughout this specification, references to "selected embodiments," "one embodiment," "an example embodiment," or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment. Thus, the appearances of the phrases "selected embodiments," "in one embodiment," "an example embodiment," or "in an embodiment" in various places throughout this specification do not necessarily refer to the same embodiment. The embodiments described herein may be combined with each other and modified to include each other's features. Furthermore, the described features, structures, or characteristics of various embodiments may be combined and modified in any suitable manner.

[0013] The illustrated embodiments will be best understood by reference to the drawings, in which like parts are designated by like numerals throughout. The following description is intended to be merely exemplary and merely sets forth certain selected embodiments of devices, systems, and processes consistent with the embodiments claimed herein.

[0014] In the field of artificial intelligence computer systems, natural language systems (such as the IBM Watson® artificial intelligence computer system and / or other natural language systems) process natural language based on knowledge acquired by the system. To process natural language, the system is trained using data obtained from a database or corpus of knowledge, but the results obtained may be incorrect or inaccurate for various reasons.

[0015] Machine learning (ML), a subset of artificial intelligence (AI), utilizes algorithms to learn from data and create predictions based on this data. AI refers to intelligence when a machine can make informed decisions that maximize the chances of success in a particular subject. More specifically, AI can learn from data sets to solve problems and provide appropriate recommendations. Cognitive computing is a blend of computer science and cognitive science. Cognitive computing utilizes self-learning algorithms to solve problems and optimize human processes using minimal data, visual recognition, and natural language processing.

[0016] At the heart of AI and related reasoning is the concept of similarity. The process of understanding natural language and objects requires reasoning in terms of relationships, which can be challenging. Structures, including static and dynamic structures, dictate determined outputs or behaviors for specific, limited inputs. More specifically, the determined outputs or behaviors are based on explicit or inherent relationships within the structures. This configuration may be sufficient for selected environments and conditions. However, it is understood that dynamic structures inherently undergo change, and correspondingly, outputs or behaviors may undergo change. Existing solutions for efficiently identifying objects, understanding natural language, and processing content in response to this identification and understanding and changes to the structures are extremely challenging to put into practical use.

[0017] A chatbot is an artificial intelligence (AI) program that simulates two-way human conversation by using pre-computed phrases and auditory or text-based signals. Chatbots are increasingly being used on electronic platforms for customer service support. In one embodiment, a chatbot can function as an intelligent virtual agent. Each chatbot experience consists of a series of communications consisting of user actions and dialogue system actions, and this experience includes characteristic behavioral patterns. It is understood in the art that chatbot interactions can be evaluated and diagnosed to identify chatbot elements that may warrant modification to improve future chatbot experiences. Such evaluations identify patterns of behavior. By learning these patterns, and more particularly, by identifying distinct characteristics of the patterns, the chatbot program can be refined or modified to improve chatbot metrics and future chatbot experiences.

[0018] Systems, computer program products, and methods automatically identify and resolve knowledge gaps. As presented and described herein, knowledge gaps are defined as contextual expressions that are expected to be equivalent but cannot be derived from one another with sufficient precision. Knowledge gaps can arise from a variety of circumstances. For example, the domain-specific concepts used to describe requirements may not be the same as the domain-specific concepts used to describe the context of operation, or the context of operation may be incomplete, because the designer neglected or omitted elements (e.g., providing details would increase the design effort of the solution), or because commonly known or accepted knowledge is assumed.

[0019] Below, two means for resolving identified knowledge gaps and providing explanations are provided and detailed: online, where end users interact with an artificial intelligence (AI) solution (e.g., a chatbot platform), and offline, where subject matter experts (SMEs) review the questions and answers generated by the system. Examples of explanations provided include, but are not limited to, enforcing existing conceptual relationships in the case of a positive response and restricting conceptual relationships in the case of a negative response. The purpose of the explanation is to reinforce the conceptual relationships presented by the solution. More specifically, domain knowledge is expanded to capture concepts and relationships, not limited to questions and answers. An exemplary type of conceptual relationship may be an equivalence conceptual relationship, such as "A" being identified as "B" or unconditionally equivalent to "B" (e.g., LAN is equivalent to "local area network"). When encountered, A or B may be substituted for the other concept without changing meaning. Another example of a type of conceptual relationship may be a contextual entailment conceptual relationship. For example, any occurrence of A brings into the context of B, such that any occurrence of A can be replaced with B and the claim would still be true, or "A" occurs simultaneously with "B" (e.g., Ethernet occurs simultaneously with wired), such that "A" implies "B" (e.g., Ethernet implies network and wired network). In example embodiments, one or more multiple-choice questions are generated to elicit possible reasons for the similarities or differences, as described in more detail below.

[0020] The chatbot platform serves as an AI dialogue interface. As shown and described herein, the chatbot platform is complemented by leveraging subject matter experts (SMEs) to provide ground truth data for bootstrapping the system. Ground truth (GT) is a term used in machine learning to refer to information provided by direct observation (e.g., empirical evidence), as opposed to information provided by inference. Attaching one or more classification tags (referred to herein as labels) to GT data provides structure and meaning to the data. Annotated GTs, or annotations, are attached to documents, or in one embodiment, document elements, to indicate the subject matter of the elements present in the document. Annotations are created and attached by annotators with different skill sets who review documents. Domain-specific relationships are collected from the chatbot platform and SMEs accordingly.

[0021] Referring to FIG. 1, a schematic diagram of an artificial intelligence (AI) platform and corresponding system (100) is shown. As shown, a server (110) is provided in communication with multiple computing devices (180), (182), (184), (186), (188), and (190) via a network connection (e.g., a computer network (105)). The server (110) is configured using a processing unit (e.g., a processor) in communication with memory via a bus. The server (110) is shown with an AI platform (150) operably coupled to a dialogue system (160) and corresponding virtual agent (162) (e.g., a chatbot), as well as a knowledge domain (170) (e.g., a data source). A visual display (130), such as a computer screen or smartphone, is provided to allow a user to interface with a representation of the virtual agent (e.g., a chatbot (162)) on the display (130). Although a visual display is shown in FIG. 1, it should be understood that the display (130) may be replaced or supplemented by other interfaces, such as an audio interface (e.g., a microphone and speaker), an audio-video interface, etc.

[0022] The AI ​​platform (150) is operatively coupled to the network (105) and supports interactions with the virtual conversational agent (162) from one or more of the computing devices (180), (182), (184), (186), (188), and (190). More specifically, the computing devices (180), (182), (184), (186), and (188) communicate with each other and with other devices or components via one or more wired and / or wireless data communication links, each of which may include one or more wires, routers, switches, transmitters, receivers, etc. In this network configuration, the server (110) and network connections (105) enable communication detection, recognition, and resolution. Other embodiments of the server (110) may be used with components, systems, subsystems, or devices, or combinations thereof, other than those shown herein.

[0023] The AI ​​platform (150) is shown herein operatively coupled to a dialogue agent (162) and configured to receive input (102) from various sources via a network (105). For example, the dialogue system (160) may receive input via the network (105) and utilize data sources (170) (also referred to herein as knowledge domains or corpora of information) to generate output or response content.

[0024] As shown, the data source (170) is configured using multiple libraries, and as an example herein, the library A (172 A ), library B (172 B ), ..., and libraries N (172 N) are shown. Each library is populated with data in the form of feedback data and ground truth data. In example embodiments, each library may be directed to a particular subject. For example, in an embodiment, the library A (172 A ) may add items related to exercise, library B (172 B ) may be populated with items targeting finance, etc. Similarly, in embodiments, data may be populated to the library based on industry. The dialogue system (160) is operatively coupled to the knowledge domains (170) and corresponding libraries.

[0025] The dialogue system (160) is a conversational AI interface for supporting communication between a virtual agent and a non-virtual agent, such as a user, which can be a human or software, and in some cases an AI virtual agent. The interactions that occur generate what are called conversations, and the content of such conversations is stored in conversation log files (also called records). Each log file records an interaction with a virtual dialogue agent (162). According to an example embodiment, each conversation log (e.g., log file) is a record of questions posed to the dialogue system (160) and corresponding answers generated by the dialogue system (160). Thus, the communication that occurs includes a dialogue within the electronic platform between the user and the virtual agent.

[0026] The dialogue system (160) is operatively coupled to a knowledge base (140) for storing records generated by the dialogue. By way of example, the knowledge base (140) is shown herein with a data structure for storing a representation of the dialogue. In this example, two data structures, DS A (142 A ) and DS B (142 B) are present. The number of data structures shown is for illustrative purposes and should not be considered limiting. Each data structure is populated with a representation of a request (e.g., a question) and a corresponding generated response or response action (e.g., an answer). As shown and described in Figures 3-8, the dialogue questions and corresponding answers are represented in models such as Abstract Meaning Representation (AMR) trees or parse trees. In example embodiments, the dialogue questions are represented in one model and the dialogue answers are represented in another model. By way of example, a first data structure DS A (142 A ) is a model that represents the question. Q,0 (142 Q,0 ), and a model representing the corresponding answers A,0 (142 A,0 ) is shown. Similarly, the second data structure DS B (142 B ) is a model that represents the question. Q,1 (142 Q,1 ), and a model representing the corresponding answers A,1 (142 A,1 ) are used. The number of models shown and described herein is for illustrative purposes and should not be considered limiting. In fact, it is contemplated that knowledge base (140) may include hundreds or thousands of interactions and corresponding interaction files or interaction data structures, with each interaction data structure including at least one request model and at least one response model.

[0027] Various computing devices (180), (182), (184), (186), (188), and (190) communicating with the network (105) may include access points to the dialogue system (162). In various embodiments, the network (105) may include local network connections and remote connections so that the AI ​​platform (150) can operate in environments of any size, including local and global (e.g., the Internet). Furthermore, the AI ​​platform (150) serves as a backend system that can make available various knowledge extracted from or represented in documents, network-accessible sources, and / or structured data sources. In this manner, processes add data to the AI ​​platform (150), which also includes an input interface for receiving requests and responding accordingly.

[0028] As shown, content may be represented by one or more models operably coupled to an AI platform (150) via a knowledge base (140). A user of the content may access the AI ​​platform (150) and an operably coupled dialogue system (160) via a network connection to the network (105) or an internet connection and may submit natural language input to the dialogue system (160), which may effectively determine an output response associated with the input from the natural language input by leveraging operably coupled data sources (170) and tools comprising the AI ​​platform (150).

[0029] The AI ​​platform (150) is shown herein with several tools for supporting a dialogue system (160) and a corresponding virtual agent (e.g., a chatbot) (162), and more particularly, with tools aimed at improving the performance of the dialogue system (160) and the virtual agent (162) experience. The AI ​​platform (150) uses several tools to interface with the virtual agent (162) and help support its performance. These tools include a dialogue manager (152), an artificial intelligence (AI) manager (154), and a director (156).

[0030] The dialogue manager (152) interfaces with the dialogue system (160) by receiving natural language (NL) associated with interactions with an automated virtual dialogue agent (162). The received natural language is shown herein added to and stored in corresponding data structures in the knowledge base (140). Each dialogue event in the knowledge base (140) includes one or more requests and one or more corresponding responses or response actions.

[0031] The AI ​​manager (154) is shown herein operatively coupled to the dialogue manager (152). The AI ​​manager (154) functions to apply received or acquired dialogue events to a learning program for knowledge gap assessment and remediation. As shown in the figure, the learning program (154) A ) is operatively coupled to the AI ​​manager (154). A ) subjects the dialogue to interpretation by generating a response in the form of an interpreted request and associated response explanation. In an example embodiment, the explanation is a rule between or bridging two or more concepts. This interpretation is used to refine the learning program (154) when knowledge gaps exist. A ) to automatically identify its presence. A) determines whether an input (also referred to herein as an input instance) contains one or more concepts that are not present (e.g., are absent) in the corresponding output instance or output behavior. The identification of knowledge gaps and instances requiring explanation is automated. Examples of situations that lead to the identification of knowledge gaps include, but are not limited to, identifying concepts in a question that do not appear to be related to concepts in the corresponding ground truth answer, identifying synonyms in the preferred answer for terms in the question, and identifying terms that represent differentiating contextual settings that are not explicitly mentioned in the question but are present in the preferred answer. In an embodiment, the learning program (154 A ) determines when input instances and corresponding output instances or output behaviors do not match. In addition to identifying knowledge gaps, the learning program (154 A ) dynamically solicits pieces of knowledge to extend the explanation of input instances and maps the requirements to output instances or output behaviors, thereby effectively bridging identified knowledge gaps. In example embodiments, the AI ​​manager (154) receives ground truth data for bootstrapping the data source (170). Similarly, in embodiments, the AI ​​manager (154) uses the explanations and solicited pieces of knowledge to enhance the data source (170). In example embodiments, the learning program (154) A ) analyzes multiple pairs of input and output instances or output behaviors (e.g., inputs and outputs of a dialogue system) to identify one or more common characteristics between two or more pairs of requests and responses. A ) and this is then leveraged to solicit pieces of knowledge to reduce or eliminate knowledge gaps.

[0032] The AI ​​manager (154) interprets the dialogue as a model representing the question. Q,0 (142 Q,0 ) and a model representing the corresponding answers A,0 (142 A,0The AI ​​manager (154) may represent the knowledge gaps in the form of a model, such as, but not limited to, a hierarchical tree (HTR) or a tree structure (KD) . This model representation allows the AI ​​manager (154) to leverage the structure and functionality of the corresponding model to compare the existing content (e.g., question and answer pair texts) and determine similarities or differences corresponding to conceptual relationships. In an example embodiment, the model is a representation of the dialogue events in the form of a subtree. The AI ​​manager (154) quantifies the knowledge gaps using a distance measurement between subgraphs containing common node labels. In an example embodiment, the measured distance corresponds to or is an indicator of the complexity of the identified knowledge gaps. In addition to the comparison, the AI ​​manager (154) may generate one or more questions for the dialogue manager (152) to communicate via the dialogue system (160) and the corresponding virtual agent (162). In an example embodiment, the dialogue manager (152) receives one or more answers to the generated questions. The received answers serve as indicators of the identified knowledge gaps and are communicated to the AI ​​manager for assessment and remediation of the knowledge gaps.

[0033] The director (156) is shown operatively coupled to the dialogue manager (152), the AI ​​manager (154), and the dialogue system (160) herein. The director (156) refines the virtual agent (162) with respect to corresponding dialogue events. This refinement is a form of facilitating and enabling the system's dialogue to gather domain-specific relationships, such as ground truth or positive feedback, from correct responses, leveraging the chatbot platform. Details of this refinement are shown and explained in FIG. 3. This refinement improves the performance of the dialogue system (160) by bridging or mitigating knowledge gaps, thereby increasing the accuracy of responses relevant to requests.

[0034] Dialogue events created or enabled by the dialogue system (160) may be processed by the IBM Watson® server (110) and corresponding artificial intelligence platform (150). The dialogue manager (152) performs analysis of the received natural language using various inference algorithms. There are likely hundreds or thousands of inference algorithms applied, each performing a different analysis (e.g., comparison). For example, some inference algorithms may examine matches of terms and synonyms within the language of the received dialogue and corresponding responses or response actions. In one embodiment, the dialogue manager (154) may process electronic communications and identify and extract features within the communications. Whether using extracted features and feature representations or an alternative platform for processing electronic records, the dialogue manager (152) processes the dialogue events in an attempt to identify and parse event and behavioral characteristics of the dialogue events. In example embodiments, behavioral characteristics include, but are not limited to, language and knowledge. In one embodiment, the platform identifies grammatical components, such as nouns, verbs, adjectives, punctuation, and so on, within the request and corresponding response or response action. Similarly, in one embodiment, one or more inference algorithms may examine temporal or spatial features within the language of the electronic record.

[0035] In some example embodiments, the server (110) may be an IBM Watson® system from International Business Machines Corporation (Armonk, NY), which has been extended using the mechanisms of example embodiments described below.

[0036] The dialogue manager (152), AI manager (154), and director (156), hereinafter collectively referred to as AI tools, are shown embodied or integrated within the artificial intelligence platform (150) of the server (110). The AI ​​tools may be implemented within a separate computing system (e.g., 190) connected to the server (110) via a network (105). When embodied, the AI ​​tools function to evaluate dialogue events, extract behavioral characteristics from requests and responses, selectively identify knowledge gaps, dynamically solicit knowledge input relevant to bridging the knowledge gaps through active explanation, and improve question transformation and knowledge representation.

[0037] In selected example embodiments, the dialogue manager (152) may be configured to apply NL processing to identify behavioral characteristics of dialogue events. For example, the NL manager (152) may perform syntactic analysis, which involves parsing the target sentence to identify grammatical terms and parts of the utterance. In one embodiment, the NL manager (152) may perform the syntactic analysis using a Slot Grammar Logic (SGL) parser. The NL manager (152) may be configured to apply one or more learning methods to match the detected content with known content to determine and assign values ​​to behavioral characteristics.

[0038] The types of information processing systems that can utilize the artificial intelligence platform (150) range from small handheld devices, such as handheld computers / cell phones (180), to large mainframe systems, such as mainframe computers (182). Examples of handheld computers (180) include personal digital assistants (PDAs), personal entertainment devices (MP4 players, portable televisions, and compact disc players). Other examples of information processing systems include pen or tablet computers (184), laptop or notebook computers (186), personal computer systems (188), and servers (190). As shown, various information processing systems can be networked together using a computer network (105). Types of computer networks (105) that can be used to interconnect various information handling systems include local area networks (LANs), wireless local area networks (WLANs), the Internet, public switched telephone networks (PSTNs), other wireless networks, and any other network topology that can be used to interconnect information handling systems. Many information handling systems include a non-volatile data store, such as a hard drive or non-volatile memory, or both. Some information handling systems may use separate non-volatile data stores (e.g., a server (190) may use a non-volatile data store (190)). A ), and the mainframe computer (182) uses a non-volatile data store (182 A )). Non-volatile data store (182 A) can be a component external to the various information handling systems or can be internal to one of the information handling systems.

[0039] The information processing system employed to support the artificial intelligence platform (150) may take a variety of forms, some of which are illustrated in FIG. 1. For example, the information processing system may take the form of a desktop, server, portable, laptop, notebook, or other form factor computer or data processing system. In addition, the information processing system may take other form factors, such as a personal digital assistant (PAD), gaming device, ATM machine, mobile phone device, communications device, or other device containing a processor and memory.

[0040] An application program interface (API) is understood in the art as intermediate software between two or more applications. With respect to the artificial intelligence platform (150) shown and described in FIG. 1, one or more APIs may be utilized to support one or more of the tools (152), (154), and (156) and their associated functionality. Referring to FIG. 2, a block diagram (200) is provided illustrating the tools (152), (154), and (156) and their associated APIs. As shown, multiple tools are incorporated within the AI ​​platform (205), including a dialogue manager (252) associated with API0 (212), an AI manager (254) associated with API1 (222), and a director (256) associated with API2 (232). Each of the APIs may be implemented in one or more languages ​​and interface specifications. API 0 (212) provides functional support for receiving and evaluating interaction events and determining behavioral characteristics; API 1 (222) provides functional support for applying received interaction events to a learning program and identifying and resolving knowledge gaps; and API 2 (232) provides functional support for using knowledge gap resolution to further refine the virtual interaction. As shown in the figure, each of APIs (212), (222), and (232) is operatively coupled to an API orchestrator (260) (otherwise referred to as an orchestrator layer), which is understood in the art to function as an abstraction layer for transparently connecting separate APIs. In embodiments, the functionality of separate APIs may be combined or combined. As such, the configuration of APIs shown herein should not be considered limiting. Accordingly, the functionality of the tools may be embodied or supported by those APIs, as shown herein.

[0041] Referring to FIG. 3, a flow chart (300) illustrating a process for enriching a corpus of reliable domain-specific semantic relations is provided. As shown and described, a virtual communication platform, referred to herein as a chatbot, serves as the foundation or platform for corpus enrichment. A question is posed or submitted to the chatbot platform (302) to solicit or initiate dialogue and response data for the chatbot (304). The chatbot uses natural language processing (NLP) in conjunction with the corpus of knowledge to generate one or more answers to the submitted question (306). In embodiments that generate multiple answers, the generated answers are referred to as alternative answers. It is understood in the art that a generated response may not provide the correct or intended answer. If the user is not satisfied with the answer and wishes to continue the dialogue, the system may collect implicit negative feedback (308) and then return to step (304) to continue the dialogue with the chatbot. In embodiments, continuing the dialogue provides implicit negative feedback. If the user wishes to end the interaction, the process continues with collecting feedback (310), which in embodiments includes collecting questions and corresponding responses (310) from the user, where final feedback is solicited (312). The final feedback may be positive or negative and corresponds to the final answer provided by the chatbot platform. Examples of feedback include "good answer" or "helpful answer" as forms of positive feedback, while "bad answer" or "unhelpful answer" are examples of negative feedback. In example embodiments, for negative feedback, multiple-choice questions such as "incorrect content" or "content not found" may be utilized to limit the scope of error.

[0042] At (314), feedback (e.g., final feedback) is stored in a repository and utilized by the system, beginning at step (316), to enhance domain knowledge. This enhancement may be in response to positive or negative feedback. Similarly, this enhancement may be via the chatbot platform or via ground truth provided by a subject matter expert (SME). Similarly, this enhancement may assess the existence of knowledge gaps, and if present, may resolve such gaps through domain knowledge enhancement in the form of interactive feedback via the chatbot (162). It is understood in the art that various methods may be utilized to generate responses for the chatbot, and thus some matches between questions (e.g., inputs) and corresponding generated answers may be partial (e.g., partially good or partially bad). While those answers may be considered good answers overall, there may be knowledge gaps that may benefit from additional information. Similarly, the system may consider an answer good when the feedback is negative, and thus may need or benefit from identifying information or knowledge to clarify the discrepancy.

[0043] Learning (also referred to herein as the learning dialogue) is shown herein as branching into a feedback dialogue (316) and a knowledge reinforcement dialogue (318). The feedback dialogue leverages the chatbot platform to facilitate and enable system interaction to gather domain-specific relationships from correct responses, such as ground truth or positive feedback. These relationships are used to expand domain knowledge. The feedback dialogue (316) is followed by the generation of an explanation prompt (320), the details of which include knowledge gap assessment, are shown and described in Figure 5. A clarification request is generated for the learning dialogue and submitted to the chatbot platform (322). Feedback in the form of an explanation response is generated via the chatbot platform and undergoes an clarification process (324). The details of the clarification process are shown and described in Figure 6. Knowledge reinforcement is performed asynchronously and is solicited from subject matter experts (SMEs), as shown in step (318). The details of knowledge reinforcement are shown and described in Figure 4. In step (318), knowledge enrichment leverages interaction channels with one or more SMEs, and the system uses the SMEs to elicit or otherwise obtain questions and corresponding answers and solicit clarification requests in the form of ground truth. In example embodiments, the ground truth is used to extend domain knowledge and increase the accuracy of data communication through the chatbot platform. The ground truth can be positive (e.g., correct answers) or negative (e.g., invalid answers). In embodiments, the ground truth is collected in a dedicated question-answer data structure, from which items are extracted and used to generate requests to one or more SMEs for clarification. Accordingly, active clarification channels are leveraged for domain knowledge expansion (also referred to herein as corpus enrichment) to refine and expand the capture of concepts and relationships for future interactions.

[0044] Referring to FIG. 4, a flow chart (400) illustrating a process for knowledge enrichment interactions is provided. As shown and described, an explanation request is generated, which leverages the corresponding schema of knowledge to develop a question for enriching domain knowledge using one or more relationships. The generation of explanation options begins with the results of the knowledge gap assessment. As shown and described in FIG. 4, matching and non-matching concepts and relationships are added to first and second data structures, respectively, which are used as inputs herein (402). The scope of knowledge enrichment is learning based on explicit feedback, which may be positive or negative and, in some cases, may be combined with implicit feedback, which may be either positive or negative. During training of the chatbot platform, one or more subject matter experts (SMEs) generate or create ground truth in the form of a set of questions and corresponding answers, and the chatbot platform learns from the ground truth how to answer posed questions and the answers (e.g., training the chatbot). The ground truth can be positive (e.g., a good answer to the posed question) or negative (e.g., an invalid answer to the posed question). In an example embodiment, the ground truth is collected in a corresponding data structure. As shown in the figure, items from the corresponding questions and answers, whether matching or not, are pulled, and a clarification request is generated (404) directed to the SME. The clarification request can be individual or one or more chunks (e.g., a CSV file) in which the questions and corresponding answers are grouped together. The SME generates a clarification response (406), which is communicated via a communication channel for processing of the clarification for analysis and collection of new semantic relationships (408), as shown and described in FIG. 7. Additionally, the SME generates the ground truth (410), which is added to the corresponding data structure.

[0045] Referring to FIG. 5, a flow chart (500) illustrating a process for generating explanations is provided. As shown and described, an initial aspect of the explanation generation process involves knowledge gap assessment to process natural language terms from questions and answers retrieved from a chatbot environment, where these terms are represented in a model for machine understanding and utilization. The process begins with input (502) in the form of question text and corresponding answer text, which in embodiments are retrieved from a chatbot platform. Using a natural language tool on the question, concepts and corresponding relationships within the question are extracted (504). In example embodiments, the tool analyzes the question and identifies components within the question, such as objects, verb-object relationships, or noun-object relationships. Information within the corresponding knowledge domain is used to identify and extract concepts (e.g., fault behavior, attributes, product management behaviors, product names, product components, etc.). Similarly, in embodiments, classifiers based on neural language models or sequence models may be employed to label which terms are related to the corresponding knowledge domain.

[0046] Representations of the components of the question (also referred to herein as requests) identified in step (504) are stored in corresponding data structures (506). For example, in embodiments, the identified components of the question are represented as Abstract Semantic Representation (AMR) trees or parse trees. AMR is a semantic representation that uses rooted, directed acyclic graphs to represent the logical meaning of a sentence. AMR associates semantic concepts with nodes on the graph, while relationships are label edges between concept nodes. In example embodiments, AMR is a structuring technique that represents the semantic representation of a sentence hierarchically, where items are layered or grouped to reduce complexity.

[0047] Following extraction in step (504) and representation in step (506), answers to the questions are analyzed (506), similar to analyzing questions. More specifically, in step (508), one or more answers to the questions are generated and analyzed. In example embodiments, the answers are retrieved from a corresponding knowledge base or knowledge domain. This analysis involves subjecting the answers to similar NL processing as questions, in which one or more components (e.g., subject or object) and corresponding properties within the question are identified. This analysis identifies the relevance of instances of extracted concepts and relationships, and in embodiments, ranks those concept and relationship instances to represent or characterize their relevance in a data structure representing the question. In example embodiments, information within the knowledge domain is used for analysis and identification. This information includes, but is not limited to, specific types of concepts or relationships, such as fault behaviors and attributes, product management actions (e.g., restart, remove, and configure), product names, and product components. Similarly, in embodiments, classifiers based on neural language models or sequence models are utilized to label which terms are relevant to the domain. In example embodiments, the extracted concepts and relationships may be arranged in a hierarchy based on the ranking. Similar to question processing, concepts in the answer text and relationships from the question related to the answer are extracted, and representations of the answer (also referred to herein as responses) components identified in step (510) are stored in corresponding data structures (512). Similar to question processing, in embodiments, the identified answer components are represented as Abstract Semantic Representation (AMR) trees or parse trees. In example embodiments, the representations of the question and answer may include multiple data structures representing multiple cognitive characteristics. Thus, both the question and the corresponding answer text undergo concept and relationship extraction.

[0048] The lists of question and answer elements added to the corresponding data structures or models in steps (504) and (508) correspond to specific selection criteria (e.g., matching or dissimilar). These lists may be generated by applying a comparison method specific to each type of feature representation. For example, two data structures may be received as input, and the output from the comparison method may generate a list of elements present in both data structures, a list of elements present only in the first data structure, and a list of elements present only in the second data structure. Differences in subtrees of a semantic or syntactic relationship graph are identified to determine subtrees that are identical or within a distance threshold for the relationship graph. For example, distances between subgraphs containing common node labels may be measured, such as the number of relationships beginning or ending at a common node.

[0049] Following step 512, the question and corresponding answer representations are compared to extract overlaps and differences 514. This comparison determines differences in subtrees of a semantic or syntactic relationship graph and determines subtrees that are identical or within a distance threshold. Distances between subgraphs containing common node labels may be measured, such as the number of relationships that begin or end at a common node, whether they are different (i.e., contain different nodes in triplet (common node, common relationship, node)), or missing (i.e., common node, relationship in only one of the trees being compared). For example, in an example embodiment, the question in this situation is "product battery must be replaced," which is represented by the following AMR representation: (o / obligate-01 :ARG2 (r / replace-01 :ARG1 (e / product :ARG1-of (e2 / batt) ) ) ) The answer text "How to replace battery on product" is expressed in the following AMR expression: (a / amr-unknown :ranner (r / replace-01 :ARG1 (b / battery) :ARG2 (e / product) Based on this example, the relationship<replace, ’action’, battery> and<batt, action, unknown> are different, the subtree representations contain a common node "replace-01" and the related subtrees are at a distance of 1. The analysis determines that "battery" is a concept of type "component" and "batt" is unknown. Therefore, this comparison leverages the data structures created in steps (506) and (512) for the comparison.

[0050] Next, it is determined (516) whether the entirety of the concepts and relationships of the extracted question are represented in the concepts and relationships of the extracted answer. A positive response to this determination is an indication that the question and answer match (e.g., are aligned) and no knowledge gap exists, and the comparison process ends (518). In an example embodiment, the evaluation in step (516) targets concepts represented within a layer of the hierarchy. A negative response to the determination in step (516) is followed by a decision to evaluate concepts and relationships related to the relationships identified in step (506) in the question and their representation in the answer (524). In an example embodiment, the evaluation in step (524) targets a different layer in the hierarchy (e.g., different concepts than those evaluated in step (516)). A positive response to the evaluation in step (524) is an indication of the evaluated question or aspects of the question determined to be present within the corresponding answer, or, in an embodiment, within the concepts for which the corresponding answer was evaluated or identified. Thus, as presented herein, the overlap of answers and questions may be aimed at evaluating one or more identified concepts.

[0051] As shown and described, the evaluation in step (524) targets concepts identified or represented in the question and may not be related to the question as a whole. Following an affirmative response to the evaluation in step (524), a determination is made (526) as to whether to continue the evaluation between the question and the answer by evaluating additional concepts represented in the question's hierarchy. In example embodiments, the user may reference one element of a concept represented in the hierarchy, but the related content pertains to related concepts in a different layer in the hierarchy. Continuing the evaluation corresponding to an affirmative response to step (526) requires additional time and, in embodiments, may disrupt the chatbot experience. A negative response to the determination in step (526) terminates the evaluation process by jumping back to the decision sequence beginning in step (518), as illustrated herein. However, a positive response to the determination in step (526) or a negative response to the determination in step (524) is followed by transforming the question text based on equivalents, such as is-a, symptom-action, and other relationships available in the domain knowledge (528). The transformation in step (528) aims to identify differences between the question and the corresponding answer. In an example embodiment, one or more language models are used to determine equivalent sentences for the question that match terms found in previous matches corresponding to the question. It is then determined (530) whether a new alternative formulation of the question has been generated. An affirmative response in step (530) is followed by a return to step (514), and a negative response is followed by a jump to the termination sequence beginning in step (518).

[0052] Upon such termination of the evaluation, shown herein as a negative response to the determinations in steps (526) and (530) and subsequent step (518), data is added to the corresponding first and second data structures. More specifically, representations of matching concepts, relationships, and graph regions are added to the first corresponding data structure (520). Similarly, representations of non-matching concepts, relationships, and graph regions are added to the second corresponding data structure (522). Although shown sequentially, in example embodiments, the additions to the first and second corresponding data structures in steps (520) and (522) may be performed in parallel or in a different order. Question elements and answer elements corresponding to specific selection criteria (e.g., matching in both representations) are added to the first data structure, and question elements and answer elements corresponding to different specific selection criteria (e.g., present in only one of the representations) are added to the second data structure. The lists are generated by applying a comparison method specific to each type of feature representation. For example, for lists of concepts, a procedure might receive two lists as input, traverse and compare the items in the lists, and output a list of elements that are present in both (e.g., matching concepts), as well as a list of elements that are present only in the first input and a list of elements that are present only in the second input (e.g., mismatching concepts). In embodiments that utilize AMR, the procedure might output pairs of subgraphs that contain the same root node label, as well as edges in the graph that contain the same label and the labels of the same neighboring nodes (e.g., matching concepts).

[0053] As shown and described in the figure, an evaluation similar to confirming the absence of a match between a question and an answer is performed to determine or identify the existence of a knowledge gap. In the example embodiment, the use of the term match covers synonym matches in addition to exact matches. The use of the term "match" is based on a comparison of the words expressed in the question with the presence of those words in the answer. The knowledge gap may be in the form of a missing connection, such as a missing synonym or an inference of a missing relationship. While users of the system provide feedback in the form of final feedback, the system analyzes the storage of questions and corresponding answers in the repository. In the example embodiment, to accommodate users with different skills and different tolerance levels, an engagement policy is provided that targets users regarding knowledge gap evaluation, so that the engagement policy selectively manages the knowledge gap learning dialogue. A partial match may be a satisfactory or good response, and the system can learn what makes a good match and learn additional relationships that contributed to the reliability of the answer. Thus, the engagement policy may learn from both collected positive and negative feedback.

[0054] The output from the knowledge gap assessment shown in FIG. 5 generates one or more explanation options for the question-answer relationship. As shown in FIGS. 4 and 5, the initial aspect is directed to an automated knowledge gap assessment, where a knowledge gap or the absence of a knowledge gap is identified. After a knowledge gap is determined or identified, the process proceeds to options for how to fill the knowledge gap. The artifacts of the explanation management system provide multiple types of knowledge, which are referred to herein as rules for mismatching with explanatory mapping artifacts, presentation templates for generating output related to user-provided explanations, and management policies to apply when two or more explanation types match. The following table (Table 1) is an example of an explanation management system artifact provided for multiple types of knowledge.

[0055] [Table 1]

[0056] [Table 2]

[0057] [Table 3] The following table (Table 2) is an example of a presentation template for generating output regarding the description provided by the user.

[0058] [Table 4] The following table (Table 3) is an example of a management policy that applies when two or more description types match.

[0059] [Table 5]

[0060] Referring to FIG. 6 , a flowchart (600) is provided illustrating a process for generating explanation options as semantic relationships between question phrases and answer phrases to enhance domain knowledge. It is understood that some aspects of the semantic relationships between questions and answers have already been identified and represented or stored in domain knowledge. For example, such prior semantic relationships may have been used to identify matches between the questions and answers in FIG. 5 . In an example embodiment, using similarity analysis to enhance domain knowledge, the differences and similarities between the questions and answers are analyzed. The process of generating explanation options uses the corresponding domain knowledge schema to develop questions to enhance the domain using relationships. The domain knowledge provides information about what is already known about equivalences and relationships.

[0061] As shown, input is provided in the form of the data structure generated in FIG. 5, which includes matching and non-matching knowledge items (602). It is understood that the knowledge items from the input may include positive or negative feedback. For positive feedback, if multiple differences exist, the differences are ranked by complexity, such as the distance between adjacent graphs, starting with smaller distances in an embodiment. Differences are filtered based on the skill of the user expected to respond to the explanation. For example, users with lower skills are not asked to explain differences represented by larger distances between graphs or subgraphs containing common node labels. Explanation choices depend on the types of relationships captured within the knowledge domain. For negative feedback, selection may be planned using different criteria. The explanation management system artifact provides the following types of knowledge: rules for associating sets of patterns with explanation types, presentation templates for generating output related to user-provided explanations, and system or management policies to be applied when two or more explanation types match the input. For each pattern in the rule that matches the input, an explanation instance is generated as output (604). The output from step (604) is a list of pattern matching instances. In an example embodiment, a pattern can be applied to additional graph regions. After step (604), system or management policies are applied to the list of explanation instances (606) to generate output in the form of final explanation instances that are presented to the user. In an example embodiment, in step (606), a set of explanation types is selected for presentation. A request for explanation is created (608) by applying one or more presentation template patterns to create output that is sent to the chatbot platform. Call details are recorded (610) and patterns are matched within the explanation management system for processing.Presentation templates are organized based on dialogue, type of explanation, explained relationships, or output sections (e.g., opening and closing statements).

[0062] Below are some examples of structured questions to ask the user in the case of an online interaction: Answer the following questions: "batt" is the opposite of "battery" (1) Equivalent (2) It is a higher-level concept (hypernym). (3) It is a lower-level concept (hyponym). (4) They are unrelated concepts

[0063] Below are example annotation questions for batch interaction: Collect terms or phrases related to the text in the "Descriptive Terms" column among several columns: (1) Equivalent (2) Higher-level concepts (hypernyms) (3) Lower-level concepts (hyponyms) (4) Other relationships or other texts

[0064] Therefore, as presented herein, a set of presentation templates in the form of questions and multiple-choice answers is used to generate the output of user-provided explanations. The questions constrain how relationships in the corresponding knowledge graph are represented.

[0065] As shown and described herein, a policy corresponds to phrasing one or more questions (e.g., prompts) to generate a synchronous response that explains the semantic relationship, or lack thereof, between the question and the answer. Similarly, in embodiments, an asynchronous response may be obtained through the use of an SME requesting clarification of an answer or batch of answers. Thus, clarification may be obtained through a synchronous channel, through the use of a chatbot platform, or through an asynchronous channel by the SME.

[0066] Referring to FIG. 7, a flowchart (700) illustrating processing and recording explanations of relationships between questions and answers is provided. Explanation processing and recording utilizes the explanation context identified in FIG. 6. Explanation responses to an explanation instance are processed (702). In an example embodiment, each explanation instance includes a corresponding explanation identifier. Each explanation identifier is a reference to metadata stored in an explanation management system that collects all details about the explanation instance. The collected details include one or more of similarities and differences between questions, answers, concepts, and relationships, and the purpose and parameters of the explanation. Using the explanation identifier or other details within the explanation response, explanation context is obtained (704), including details about the purpose and parameters of the explanation. The selected explanation choice and explanation metadata in the explanation response are then used to determine the type of relationship identified by the user (706). Next, it is determined whether the user selected "I don't know" as a response (708). An affirmative response at step (708) is followed by saving the explanation for review by the SME (710), and a negative response at step (708) is followed by determining whether the user selected "irrelevant concept" as the response (712). An affirmative response to the determination at step (712) is followed by determining whether another explanation request exists for the current context (714). An affirmative response to the determination at step (714) is followed by returning to step (706), and a negative response ends evaluation of the explanation.

[0067] A negative response to the determination in step (712) is an indication that knowledge artifacts will be created from the relationships indicated by the user. A mapping is created (716) to associate explanation types (e.g., relationship types and concept types) with one or more domain knowledge components and artifact types, followed by an update operation (718) on the knowledge domain for each knowledge artifact generated from the relationships indicated by the user.

[0068] Referring to FIG. 8, a block diagram (800) illustrating an exemplary relationship-knowledge artifact map is provided. As shown, there are two columns within the map, including a relationship-concept type column (810) and a domain knowledge component-artifact column (820). In this example, equivalence relationships (812) involving verbs, nouns, and phrases, such as equivalent meanings expressed in the form of verbs, nouns, or phrases, are integrated as synonyms in the search engine (822). Similarly, "is-a" or hypernym relationships (814) are integrated in synonyms expanded by the search engine (824), cause-effect or negation relationships (816) are integrated as synonyms expanded by the knowledge graph or search engine (826), and symptom-treatment relationships (818) are knowledge graph artifacts (828). For each knowledge artifact generated, an update operation to the domain knowledge is initiated (718). The update actions in step (718) can be applied immediately or collected and applied in batches during predefined system maintenance windows. Thus, as presented herein, knowledge gaps are subjected to analysis to process the descriptions of the knowledge gaps.

[0069] Returning to FIG. 3 , active learning from semantic relationship explanations is performed as shown and described therein. As shown in the figure, knowledge enrichment branches into a chatbot platform (316) for learning dialogue and ground truth solicitation (318) via one or more SMEs. In example embodiments, if the correspondence between the question and the answer is weak (e.g., not strong), the system generates one or more candidate explanation relationships that can enforce the correspondence and utilizes an interface (e.g., a chatbot) to restrict relevance, for example, as shown and described in FIG. 5 . The learning dialogue enables or supports online evaluation of the explanation. In example embodiments, feedback (e.g., implicit or explicit feedback, positive or negative feedback, etc.) regarding the goodness of the answer may be used to trigger learning from the explanation. Similarly, in embodiments, policies may be used to determine when to trigger the dialogue for clarification request in step (318) to control the impact on user satisfaction. An exemplary policy may be embodied for negative feedback only, or another exemplary policy may relate to the user's skill level (e.g., beginner). As shown, the learning dialogue (322) generates an explanatory response (324), which undergoes processing and recording of an explanation of the relationship between the question and the answer (324), as shown and described in FIG. 3. Steps (324) and (318) are followed by processing of the explanatory response, and the identified relationship is stored in domain knowledge (326). Accordingly, the relationship between the target phrase and the missing information and the corresponding artifact are identified and stored in the knowledge base.

[0070] As shown and described in Figures 1-8, computer systems, program products, and methods are provided for comparing question and answer pairs, and particularly their text, to determine similarities and differences with respect to conceptual relationships related to how the answers relate to or refer to expected relationships derived from the questions. The similarities and differences are leveraged to generate specific multiple-choice questions regarding possible reasons for the differences. The questions are presented to an AI dialogue platform (e.g., a chatbot platform) to extract relationships from correct responses, such as ground truth or positive feedback. The extracted relationships, and in embodiments, new relationships (e.g., explanatory information), are identified and used to extend domain knowledge, thereby expanding domain knowledge using the captured concepts and relationships.

[0071] The embodiments shown and described herein may be in the form of a computer system for use with an intelligent computing platform to enhance domain knowledge. Aspects of the tools 152, 154, and 156 and their associated functionality may be embodied in a computer system / server at a single location or, in embodiments, configured within a cloud-based system sharing computing resources. Referring to FIG. 9, a block diagram 900 is provided illustrating an example of a computer system / server 902 (hereinafter referred to as a host 902) in communication with a cloud-based support system 910 to implement the systems, tools, and processes described above in FIGS. 1-8. In an embodiment, the host 902 is a node in a cloud computing environment. The host 902 is operational with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or combinations thereof, that may be suitable for use with the host (902) include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, microcomputer systems, mainframe computer systems, and file systems that include any of these systems, devices, and the like (e.g., distributed storage environments and distributed cloud computing environments).

[0072] The host (902) may be described in the general context of instructions executable by a computer system, such as program modules being executed by the computer system. Typically, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The host (902) may be executed in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0073] As shown in FIG. 9, the host (902) is depicted in the form of a general-purpose computing device. Components of the host (902) may include, but are not limited to, one or more processors or processing units (904) (e.g., hardware processors), system memory (906), and a bus (908) coupling various system components, including the system memory (906), to the processor (904). The bus (908) may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. Examples of such architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnects (PCI) bus. The host 902 typically includes a variety of computer system readable media, which may be any available media that can be accessed by the host 902, including volatile and nonvolatile media, removable and non-removable media.

[0074] The system memory (906) may include computer-system-readable media in the form of volatile memory, such as random access memory (RAM) (930) and / or cache memory (932). By way of example only, a storage system (934) may be provided for reading from and writing to non-removable, non-volatile magnetic media (not shown, typically referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to removable, non-volatile magnetic disks (e.g., "floppy disks"), and an optical disk drive may be provided for reading from and writing to removable, non-volatile optical disks, such as CD-ROMs, DVD-ROMs, or other optical media. In such examples, each may be connected to the bus (908) by one or more data media interfaces.

[0075] For example, a program / utility (940) including a set of (at least one) program modules (942) may be stored in system memory (906), including, but not limited to, an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or a combination thereof, may include an implementation of a network environment. The program modules (942) generally perform the functions and / or methods of the embodiments, dynamically interpreting and understanding requests and operational descriptions, effectively extending corresponding domain knowledge. For example, the set of program modules (942) may include tools (152), (154), and (156) shown in FIG. 1.

[0076] The host (902) may communicate with one or more external devices (914), such as a keyboard, a pointing device, a display (924), one or more devices that allow a user to interact with the host (902), or any device (e.g., a network card, a modem, etc.) that allows the host (902) to communicate with one or more other computing devices, or a combination thereof. Such communication may occur through an input / output (I / O) interface (922). Additionally, the host (902) may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, through a network adapter (920). As shown, the network adapter (920) communicates with the other components of the host (902) via a bus (908). In an embodiment, multiple nodes of a distributed file system (not shown) communicate with a host (902) via an I / O interface (922) or via a network adapter (920). Although not shown, it should be understood that other hardware and / or software components may be used with the host (902), including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0077] In this document, the terms "computer program medium," "computer usable medium," and "computer readable medium" are used to generally refer to media such as main memory (906), including RAM (930), cache (932), and storage systems (934), such as removable storage drives and hard disks installed in hard disk drives.

[0078] Computer programs (also called computer control logic) are stored in memory 906. The computer programs may be received via a communications interface, such as a network adapter 920. When executed, such computer programs enable the computer system to perform the features of the present embodiments described herein. In particular, when executed, the computer programs enable the processing unit 904 to perform the functions of the computer system. Thus, such computer programs represent the controller of the computer system.

[0079] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, dynamic or static random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), magnetic storage devices, portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, punch cards, or mechanically encoded devices such as ridge structures in grooves in which instructions are recorded, and any suitable combination thereof. As used herein, a computer-readable storage medium should not itself be construed as a transitory signal such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted over a wire.

[0080] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device over a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). This network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage on a computer-readable storage medium within each computing / processing device.

[0081] The computer-readable program instructions for carrying out the operations of the present embodiments may be source or object code written in any combination of one or more programming languages, including assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java®, Smalltalk®, C++, and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and on a remote computer, or entirely on a remote computer or server or cluster of servers. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to customize the electronic circuitry to perform aspects of the embodiments.

[0082] The functional tools described herein are labeled as managers. Managers may be implemented in programmable hardware devices, such as field programmable gate arrays, programmable array logic, programmable logic devices, etc. Managers may also be implemented in software for processing by various types of processors. For example, an identified manager of executable code may comprise one or more physical or logical blocks of computer instructions organized, for example, as objects, procedures, functions, or other constructs. Nevertheless, the executable files of an identified manager need not be physically located together and may include heterogeneous instructions stored in different locations that, when logically combined together, constitute the manager and achieve the manager's specified purpose.

[0083] In practice, a manager of executable code can be a single instruction or many instructions, and may be distributed across multiple different code segments, among different applications, and across multiple memory devices. Similarly, manipulable data, as identified and represented herein within a manager, may be embodied in any suitable form and organized within any suitable type of data structure. Manipulable data may be collected as a single data set or distributed across different locations, including different storage devices, and may exist at least in part as electronic signals on a system or network.

[0084] Referring now to FIG. 10, an exemplary cloud computing network (1000) is shown. As shown, the cloud computing network (1000) includes a cloud computing environment (1050) that includes one or more cloud computing nodes (1010) with which local computing devices used by cloud consumers may communicate. Examples of these local computing devices include, but are not limited to, a personal digital assistant (PDA) or mobile phone (1054A), a desktop computer (1054B), a laptop computer (1054C), or an automotive computer system (1054N), or combinations thereof. Individual nodes within the nodes (1010) may further communicate with each other. The nodes 1010 may be physically or virtually grouped together in one or more networks (not shown), such as a private cloud, community cloud, public cloud, or hybrid cloud, as previously described herein, or any combination thereof. This allows the cloud computing environment (1000) to provide an infrastructure, platform, and / or SaaS that eliminates the need for cloud consumers to maintain resources on local computing devices. The types of computing devices (1054A-N) shown in Figure 10 are intended as examples only, and it is understood that the cloud computing environment (1050) can communicate with any type of computer-controlled device via any type of network and / or network-addressable connection (e.g., a connection using a web browser).

[0085] Referring now to Figure 11, there is shown a set of functional abstraction layers (1100) provided by the cloud computing network of Figure 10. It should be understood in advance that the components, layers, and functions shown in Figure 11 are intended to be illustrative only and are not limiting to the embodiments. As shown, a hardware and software layer (1110), a virtualization layer (1120), a management layer (1130), and a workload layer (1140), and corresponding functions, are provided.

[0086] The hardware and software layer (1110) includes hardware and software components. Examples of hardware components include mainframes (e.g., IBM® zSeries® systems), RISC (Reduced Instruction Set Computer) architecture-based servers (e.g., IBM pSeries® systems), IBM xSeries® systems, IBM BladeCenter® systems, storage devices, networks, and network components. Examples of software components include network application server software (e.g., IBM WebSphere® application server software) and database software (e.g., IBM DB2® database software). (IBM, zSeries, pSeries, xSeries, BladeCenter, WebSphere, and DB2 are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide.)

[0087] The virtualization layer (1120) comprises an abstraction layer that can provide virtual entities such as virtual servers, virtual storage, virtual networks including virtual private networks, virtual applications and operating systems, and virtual clients.

[0088] In an example, the management layer (1130) may provide functionality for resource provisioning, metering and pricing, a user portal, service layer management, and SLA planning and execution. Resource provisioning dynamically procures computing and other resources used to execute tasks within the cloud computing environment. Metering and pricing tracks costs as resources are utilized within the cloud computing environment and generates and sends bills for the use of those resources. In one example, these resources may include application software licenses. Security verifies the identity of cloud consumers and tasks and protects data and other resources. A user portal provides consumers and system administrators with access to the cloud computing environment. Service layer management allocates and manages cloud computing resources to meet required service layers. Service level agreement (SLA) planning and execution proactively provisions and procures cloud computing resources in anticipation of upcoming demand, in accordance with SLAs.

[0089] The workload layer (1140) represents examples of functionality available in a cloud computing environment. Examples of workloads and functionality provided by this layer include, but are not limited to, mapping and navigation, software development and lifecycle management, virtual classroom education delivery, data analytics processing, transaction processing, and natural language enhancements.

[0090] While particular embodiments of the present invention have been shown and described, it will be apparent to those skilled in the art, based on the contents of this specification, that changes and modifications may be made without departing from the embodiments and their broader aspects. Accordingly, the appended claims encompass within their scope all such changes and modifications that are within the true spirit and scope of the embodiments. It should be further understood that the embodiments are defined solely by the appended claims. Where a specific number of introduced claim elements is intended, such intent will be expressly recited in the claims; it will be understood by those skilled in the art that, absent such recitation, no such limitation exists. As an aid to understanding with respect to non-limiting examples, the following appended claims include the use of the introductory phrases "at least one" and "one or more" to introduce claim elements. However, the use of such phrases should not be construed as meaning that the introduction of a claim element by the indefinite article "a" or "an" limits any particular claim containing such introduced claim element to embodiments containing only one such element, even if that same claim also contains the introductory phrase "one or more" or "at least one" and an indefinite article such as "a" or "an," and the same applies to the use of definite articles in the claims. As used herein, the term "and / or" means either or both (or one or any combination or all of the terms expressed and referenced).

[0091] The present embodiments may be systems, methods, or computer program products, or combinations thereof. In addition, selected aspects of the present embodiments may take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments combining software or hardware aspects or both, all of which may be referred to generally herein as "circuits," "modules," or "systems." Furthermore, aspects of the present embodiments may take the form of a computer program product embodied in a computer-readable storage medium containing computer-readable program instructions for causing a processor to execute aspects of the present embodiments. When so embodied, the disclosed systems, methods, or computer program products, or combinations thereof, function to support natural language enrichment.

[0092] Aspects of the present embodiments are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to the embodiments. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, are implemented by computer-readable program instructions.

[0093] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to create a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium, capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0094] Furthermore, the computer-readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device to generate a computer-implemented process, thereby causing a series of operable steps to be performed on the computer, other programmable apparatus, or other device, such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0095] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions shown in the blocks may occur in an order different from that shown in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks included in the block diagrams and / or flowchart diagrams, are implemented by a special-purpose hardware-based system that performs the specified function or operation or executes a combination of special-purpose hardware and computer instructions.

[0096] Although particular embodiments have been described herein for purposes of illustration, it will be understood that various modifications may be made without departing from the spirit and scope of the embodiments. Accordingly, the scope of protection for the embodiments is limited only by the following claims and their equivalents.

Claims

1. 1. A computer system, comprising: a processor operatively coupled to the memory; an artificial intelligence (AI) platform in communication with the processor, the AI ​​platform including tools for managing performance of a dialogue system, the tools comprising: a dialogue manager for receiving natural language (NL) associated with a dialogue with an automated virtual dialogue agent of the dialogue system, the NL comprising one or more dialogue events, each dialogue event comprising one or more input instances and one or more corresponding output instances or output actions; an AI manager operatively coupled to the dialogue manager for applying the received dialogue events to a learning program, the applying comprising: interpreting the one or more input instances and the output instances or output behaviors; identifying a knowledge gap associated with one or more of the interpreted instances or the interpreted output behavior; dynamically generating one or more alternative knowledge items to bridge the identified knowledge gap; The AI ​​manager includes the learning program for executing a director for improving the automated virtual dialogue agent with respect to the dialogue events, the improvement corresponding to the bridged knowledge gap to improve performance of the dialogue system; and It is equipped with The AI ​​manager: The model represents the interaction events as a subtree representation and measures the distance between subgraphs containing common node labels. It is further structured as follows: the measured distance corresponds to the complexity of the identified knowledge gap; The computer system.

2. 1. A computer system, comprising: a processor operatively coupled to the memory; an artificial intelligence (AI) platform in communication with the processor, the AI ​​platform including tools for managing performance of a dialogue system, the tools comprising: a dialogue manager for receiving natural language (NL) associated with a dialogue with an automated virtual dialogue agent of the dialogue system, the NL comprising one or more dialogue events, each dialogue event comprising one or more input instances and one or more corresponding output instances or output actions; an AI manager operatively coupled to the dialogue manager for applying the received dialogue events to a learning program, the applying comprising: interpreting the one or more input instances and the output instances or output behaviors; identifying a knowledge gap associated with one or more of the interpreted instances or the interpreted output behavior; dynamically generating one or more alternative knowledge items to bridge the identified knowledge gap; The AI ​​manager includes the learning program for executing a director for improving the automated virtual dialogue agent with respect to the dialogue events, the improvement corresponding to the bridged knowledge gap to improve performance of the dialogue system; and It is equipped with The AI ​​manager: The interpretation of the one or more input instances represents the dialogue events in a model and utilizes the model to compare question and answer text pairs and determine similarities and differences with respect to conceptual relationships. further configured as follows: The computer system.

3. The learning program: analyzing a plurality of pairs of inputs and outputs of the dialogue system, identifying one or more common features among the plurality of pairs, and utilizing the identified one or more common features to solicit a portion of knowledge; further configured as follows:

3. A computer system according to claim 1 or 2.

4. The AI ​​manager: The explanation and part of the requested knowledge are used to update the corresponding domain knowledge. further configured as follows:

4. The computer system of claim 3.

5. The AI ​​manager: receiving ground truth data for bootstrapping the domain knowledge; further configured as follows:

5. The computer system of claim 4.

6. 6. The computer system of claim 4, wherein the explanation is a relationship between two or more concepts.

7. The AI ​​manager: The interpretation of the one or more input instances represents the dialogue events in a model and utilizes the model to compare question and answer text pairs and determine similarities and differences with respect to conceptual relationships. further configured as follows: A computer system according to claim 1 or any one of claims 3 to 6 when claim 1 is recited.

8. The AI ​​manager: Dynamically soliciting a portion of knowledge that extends the explanation generates one or more questions from the determined similarities and differences, presents the generated one or more questions via the dialogue system, and receives at least one answer to the generated one or more questions. It is further structured as follows: the answers serve as indicators of the identified knowledge gaps; A computer system according to claim 7 when claim 4 is recited.

9. The AI ​​manager: identifying the knowledge gaps to determine when the one or more input instances include one or more concepts that are not present in the one or more corresponding output instances of an operation; further configured as follows:

9. A computer system according to any one of claims 1 to 8.

10. The AI ​​manager: and requesting a selection of at least one of the one or more generated alternative knowledge items. It is further structured as follows: the selection bridges the identified knowledge gap; 10. A computer system according to any one of claims 1 to 9.

11. 1. A computer system, comprising: an artificial intelligence (AI) platform in communication with a processor, the AI ​​platform including tools for managing performance of a dialogue system, the tools including: a dialogue manager for receiving natural language (NL) associated with a dialogue with an automated virtual dialogue agent of the dialogue system, the NL comprising one or more dialogue events, each dialogue event comprising one or more input instances and one or more corresponding output instances; an AI manager operatively coupled to the dialogue manager for applying the received dialogue events to a learning program, the applying comprising: identifying a knowledge gap between the one or more input instances and the one or more corresponding output instances; dynamically generating one or more alternative knowledge items to bridge the identified knowledge gap; The AI ​​manager includes the learning program for executing a director for improving the automated virtual dialogue agent, the improvement corresponding to the bridged knowledge gap; and It is equipped with The AI ​​manager The model represents the interaction events as a subtree representation and measures the distance between subgraphs containing common node labels. It is further structured as follows: the measured distance corresponds to the complexity of the identified knowledge gap; The computer system.

12. 1. A computer system, comprising: an artificial intelligence (AI) platform in communication with a processor, the AI ​​platform including tools for managing performance of a dialogue system, the tools including: a dialogue manager for receiving natural language (NL) associated with a dialogue with an automated virtual dialogue agent of the dialogue system, the NL comprising one or more dialogue events, each dialogue event comprising one or more input instances and one or more corresponding output instances; an AI manager operatively coupled to the dialogue manager for applying the received dialogue events to a learning program, the applying comprising: identifying a knowledge gap between the one or more input instances and the one or more corresponding output instances; dynamically generating one or more alternative knowledge items to bridge the identified knowledge gap; The AI ​​manager includes the learning program for executing a director for improving the automated virtual dialogue agent, the improvement corresponding to the bridged knowledge gap; and It is equipped with The AI ​​manager Interpreting the one or more input instances represents the dialogue events in a model and utilizes the model to compare question and answer text pairs and determine similarities and differences with respect to conceptual relationships. further configured as follows: The computer system.

13. 1. A computer-implemented method for performance of a dialogue system, the computer-implemented method comprising: receiving, by a processor of a computing device, natural language (NL) associated with a dialogue with an automated virtual dialogue agent of a dialogue system, said NL including one or more dialogue events, each dialogue event including one or more input instances and one or more corresponding output instances or output actions; applying, by the processor, the received interaction events to an artificial intelligence (AI) platform including a learning program; interpreting the one or more input instances and the output instances or output behaviors; identifying a knowledge gap associated with one or more of the interpreted instances or behaviors or the interpreted output behavior; said applying including dynamically generating one or more alternative knowledge items to bridge said identified knowledge gap; refining the automated virtual dialogue agent with respect to the dialogue events corresponding to the bridged knowledge gap to improve performance of the dialogue system; Including, The model further comprises representing the interaction events in a sub-tree representation and measuring the distance between sub-graphs containing common node labels, the measured distance corresponding to the complexity of the identified knowledge gap. The computer-implemented method.

14. 1. A computer-implemented method for performance of a dialogue system, the computer-implemented method comprising: receiving, by a processor of a computing device, natural language (NL) associated with a dialogue with an automated virtual dialogue agent of a dialogue system, said NL including one or more dialogue events, each dialogue event including one or more input instances and one or more corresponding output instances or output actions; applying, by the processor, the received interaction events to an artificial intelligence (AI) platform including a learning program; interpreting the one or more input instances and the output instances or output behaviors; identifying a knowledge gap associated with one or more of the interpreted instances or behaviors or the interpreted output behavior; said applying including dynamically generating one or more alternative knowledge items to bridge said identified knowledge gap; refining the automated virtual dialogue agent with respect to the dialogue events corresponding to the bridged knowledge gap to improve performance of the dialogue system; Including, interpreting the one or more input instances further includes representing the dialogue events with a model and utilizing the model to compare question and answer text pairs to determine similarities and differences with respect to conceptual relationships. The computer-implemented method.

15. The learning program: analyzing a plurality of pairs of inputs and outputs of the dialogue system, identifying one or more common features among the plurality of pairs, and utilizing the identified one or more common features to solicit a portion of knowledge; further configured as follows:

15. A computer-implemented method according to claim 13 or 14.

16. The computer-implemented method of claim 15 , further comprising using the explanation and the portion of the requested knowledge to enhance corresponding domain knowledge.

17. The computer-implemented method of claim 16 , further comprising receiving ground truth data for bootstrapping the domain knowledge.

18. 18. The computer-implemented method of claim 13 or any one of claims 15 to 17 when citing claim 13, wherein interpreting the one or more input instances further comprises representing the dialogue events in a model and utilizing the model to compare question and answer text pairs to determine similarities and differences with respect to conceptual relationships.

19. 19. The computer-implemented method of claim 18 when citing claim 16, wherein dynamically requesting portions of knowledge that expand the explanation further includes generating one or more questions from the determined similarities and differences, presenting the generated one or more questions via the dialogue system, and receiving at least one answer to the generated one or more questions, the answer serving as an indicator of the identified knowledge gap.

20. A computer program product for causing a processor to carry out the steps of the method according to any one of claims 13 to 19.

Images