Natural language enrichment method using action explanations, computer system and program product

Abstract

A computer system, computer program product, and computer-implemented method for improving performance of a dialog system employing an automated virtual dialog agent. The method involves receiving a natural language request and generating a corresponding response with an automated virtual agent, automatically identifying and resolving a corresponding knowledge gap between the request and the response, and refining the automated virtual agent with the resolved knowledge gap.

Description

Background Technology

[0001] This embodiment relates to virtual dialogue systems employing automated virtual dialogue agents such as "chatbots," as well as related computer program products and computer-implemented methods. In some exemplary embodiments, knowledge gaps between one or more requests and corresponding expected responses are identified and resolved, with the solution aimed at bridging or minimizing these knowledge gaps to improve the performance of the automated virtual dialogue agent.

[0002] A chatbot is a computer program that uses artificial intelligence (AI) as a platform to facilitate transactions between an automated virtual dialogue agent and a user, typically a consumer. Transactions can involve product sales, customer service, information retrieval, or other types of transactions. Chatbots interact with users through dialogue, which is typically text-based (e.g., online or via text) or auditory (e.g., via telephone). The function of a chatbot, as known in the art, is a question-and-answer component between the user and the AI ​​platform. The quality of questions and answers derives from the quality of question understanding, question transformation, and answer parsing. A frequent cause of errors often found in failing to find a corresponding response to a question is the lack of knowledge to effectively transform the question into an equivalent knowledge representation mapped to the answer. For example, the lack of synonyms or conceptual relationships limits the AI ​​platform's ability to determine whether a question is equivalent to or related to a known question for which an answer is available. Summary of the Invention

[0003] The embodiments include systems, computer program products, and methods for improving the performance of dialogue systems, and in specific embodiments, the improvement is directed to dynamically requesting input of relevant knowledge for proactive interpretation that bridges knowledge gaps.

[0004] In one aspect, a system is provided for use with a computer system including a processing unit (e.g., a processor) operatively coupled to memory and an artificial intelligence (AI) platform communicating with the processing unit. The AI ​​platform is configured with tools to guide the execution of the operatively coupled dialogue system. These tools include a dialogue manager, an AI manager, and a supervisor. The dialogue manager receives and processes natural language (NL) relating to interactions with an automated virtual dialogue agent of the dialogue system. NL includes one or more input instances and one or more corresponding output instances or dialogue events in the form of output actions. The AI ​​manager applies the dialogue events to a learning process to interpret one or more requests, identify knowledge gaps, and dynamically bridge knowledge gaps. The supervisor optimizes the automated virtual dialogue agent to correspond to the bridged knowledge gaps, with the optimization aimed at improving the performance of the dialogue system.

[0005] On the other hand, a computer program product is provided for improving the performance of a virtual dialogue agent system. The computer program product includes a computer-readable storage medium having program code embodied therein. The program code is executable by a processor to direct the execution of an operationally coupled dialogue system. The program code is used to receive and process natural language (NL) relating to interactions with an automated virtual dialogue agent of the dialogue system. NL includes one or more input instances and one or more corresponding output instances or dialogue events in the form of output actions. The program code is also used to apply dialogue events to a learning process to interpret one or more requests, identify knowledge gaps, and dynamically bridge knowledge gaps. Program code is also provided to optimize the automated virtual dialogue agent to correspond to the bridged knowledge gaps, with the optimization aimed at improving the performance of the dialogue system.

[0006] In another aspect, a computer-implemented method for improving the performance of a dialogue system is provided. The method includes: receiving and processing natural language (NL) in relation to interactions with an automated virtual dialogue agent by a processor of a computing device. NL includes one or more input instances and one or more corresponding output instances or dialogue events in the form of output actions. The dialogue events are applied to a learning process to interpret the one or more input instances, identify knowledge gaps, and dynamically bridge the knowledge gaps. The automated virtual dialogue agent undergoes optimization commensurate with the bridged knowledge gaps, which aims to improve the performance of the dialogue system.

[0007] In a further aspect, a computer system with an artificial intelligence (AI) platform communicating with a processor is provided. The AI ​​platform is configured with tools to guide the execution of an operationally coupled dialogue system. These tools include a dialogue manager, an AI manager, and a supervisor. The dialogue manager receives and processes natural language (NL) related to interactions with an automated virtual dialogue agent of the dialogue system. The AI ​​manager applies dialogue events to a learning process to interpret one or more input instances, identify knowledge gaps, and dynamically bridge knowledge gaps. The supervisor optimizes the automated virtual dialogue agent to correspond to the bridged knowledge gaps, with the optimization aimed at improving the performance of the dialogue system.

[0008] These and other features and advantages will become apparent from the following detailed description of the present exemplary embodiments in conjunction with the accompanying drawings. Attached Figure Description

[0009] The accompanying drawings, which are referenced herein, form part of the specification and are incorporated herein by reference. The features shown in the drawings are merely illustrative of some embodiments, and not of all embodiments, unless explicitly indicated otherwise.

[0010] Figure 1 A system diagram illustrating an artificial intelligence platform computing system in a network environment is presented.

[0011] Figure 2 Depicting as shown Figure 1 The diagram shown illustrates the artificial intelligence platform tools and their associated application programming interfaces (APIs).

[0012] Figure 3 A flowchart depicts an embodiment of a method for enriching a corpus of trusted domain-specific semantic relations.

[0013] Figure 4 A flowchart illustrating an embodiment of a knowledge-rich interaction method is provided.

[0014] Figure 5 A flowchart illustrating an embodiment of the method for generating explanations is depicted.

[0015] Figure 6 A flowchart depicts an embodiment of a method for generating explanatory options as semantic relationships between question and answer phrases to enrich domain knowledge.

[0016] Figure 7 A flowchart illustrating the relationship between processing and recording questions and answers is provided.

[0017] Figure 8 A diagram illustrating example relationships—knowledge artifacts—mappings is provided.

[0018] Figure 9 A block diagram illustrating an example of a cloud-based supporting computer system / server is provided to achieve the above-mentioned... Figures 1-8 The system and process described.

[0019] Figure 10 A block diagram illustrating a cloud computing environment is depicted.

[0020] Figure 11 A block diagram illustrating a set of functional abstraction model layers provided by a cloud computing environment is depicted. Detailed Implementation

[0021] It will be readily understood that, as generally described and illustrated in the accompanying drawings, the components of this embodiment can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of embodiments of the apparatus, system, method, and computer program product of this embodiment, as presented in the drawings, is not intended to limit the scope of the claimed embodiments, but is merely representative of selected embodiments.

[0022] Throughout this specification, references to "selective embodiment," "one embodiment," "exemplary embodiment," or "embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. Therefore, the phrases "selective embodiment," "in one embodiment," "in an exemplary embodiment," or "in an embodiment" appearing throughout this specification do not necessarily refer to the same embodiment. The embodiments described herein can be combined with each other and modified to include features of each other. Furthermore, the features, structures, or characteristics described in the various embodiments can be combined and modified in any suitable manner.

[0023] The illustrated embodiments will be better understood by referring to the accompanying drawings, in which the same components are always designated by the same reference numerals. The following description is intended to be illustrative only and shows only some selected embodiments of devices, systems, and processes consistent with the embodiments claimed herein.

[0024] In the field of artificial intelligence computer systems, natural language systems (such as IBM Watson) ® Artificial intelligence computer systems or other natural language systems process natural language based on knowledge acquired by the system. To process natural language, the system can be trained with data derived from databases or corpora of knowledge; however, for various reasons, the results may be incorrect or inaccurate.

[0025] Machine learning (ML), a subset of artificial intelligence (AI), uses algorithms to learn from data and create predictions based on that data. AI refers to the intelligence of machines when they can make decisions based on information, maximizing the chances of success on a given subject. More specifically, AI can learn from datasets to solve problems and provide relevant recommendations. Cognitive computing is a hybrid of computer science and cognitive science. Cognitive computing utilizes self-learning algorithms that employ data minimization, visual recognition, and natural language processing to solve problems and optimize human processes.

[0026] At the heart of AI and related reasoning lies the concept of similarity. Understanding natural language and objects requires reasoning from a relational perspective, which can be challenging. Structures, including static and dynamic structures, define predetermined outputs or actions for a given set of inputs. More specifically, the predetermined outputs or actions are based on expressions or inherent relationships within the structure. This arrangement can be satisfactory for choosing contexts and conditions. However, it should be understood that dynamic structures are inherently subject to change, and the outputs or actions can change accordingly. Existing solutions for effectively identifying objects and understanding natural language, as well as handling changes in the content and structure of the identified and understood content, are extremely difficult to implement in practice.

[0027] Chatbots are artificial intelligence (AI) programs that simulate interactive human conversation using pre-computed phrases and auditory or text-based signals. Chatbots are increasingly used in electronic platforms for customer service support. In one embodiment, a chatbot can act as an intelligent virtual agent. Each chatbot experience consists of a set of communications, comprising user actions and dialogue system actions, where the experience exhibits differentiated behavioral patterns. It is understood in the art that chatbot conversations can be evaluated and diagnosed to identify elements of the chatbot that can be redefined to improve future chatbot experiences. Such evaluations identify behavioral patterns. By studying these patterns, and more specifically by identifying the distinct characteristics of these patterns, chatbot programs can be optimized or modified to improve chatbot metrics and future chatbot experiences.

[0028] A system, computer program product, and method for automatically identifying and resolving knowledge gaps. As shown and described herein, a knowledge gap is defined as a contextual representation that is expected to be equivalent but cannot be derived from each other with sufficient precision. Knowledge gaps may arise from different scenarios. For example, the domain-specific concept used to describe a request may differ from the concept used to describe the action context, or the action context may be incomplete because the designer missed or omitted specification of elements, such as providing details that increase the solution design effort, or because of assumptions about well-known or accepted knowledge.

[0029] Two approaches, described in detail below, are provided to address the identified knowledge gaps and provide explanations thereof: online interaction with end-users who interact with artificial intelligence (AI) solutions (such as chatbot platforms), and offline interaction with subject matter experts (SMEs) to review questions and answers generated by the system. Examples of the explanations provided include, but are not limited to, enforcing existing conceptual relationships for positive responses and limiting conceptual relationships within the context for negative responses. The purpose of the explanations is to enrich the conceptual relationships presented in the solution. More specifically, domain knowledge is extended to capture concepts and relationships, not just questions and answers. Example types of conceptual relationships can be equivalent conceptual relationships, such as “A” being identified or unconditionally equivalent to “B,” e.g., LAN being equivalent to “local area network.” If any of these occur, A or B can be replaced by other concepts without changing the meaning. Another example of a type of conceptual relationship can be a context-implied conceptual relationship. For example, “A” and “B” co-occur, e.g., Ethernet co-occurs with wires, such that any occurrence of A generates the context of B, or “A” implies “B,” e.g., Ethernet implies network and wired network, such that any occurrence of A can be replaced by B and the assertion remains true. In an exemplary embodiment and in detail below, one or more multiple-choice questions are generated to elicit possible reasons for similarity or difference.

[0030] Chatbot platforms act as AI interaction interfaces. As shown and described in this paper, subject matter experts (SMEs) are used to supplement chatbot platforms to provide ground truth data to guide the system. Ground truth (GT) is a term used in machine learning that refers to information provided through direct observation, such as empirical evidence, as opposed to information provided through inference. Attaching one or more categorical labels (referred to as labels in this paper) to GT data provides structure and meaning to the data. Annotated GTs or labels are attached to documents or to an instance element of a document and indicate the subject matter of the elements present within the document. Annotations are created and attached by annotators from different skill groups who viewed the document. Thus, domain-specific relationships are collected from the chatbot platform and SMEs.

[0031] refer to Figure 1 A schematic diagram of an artificial intelligence (AI) platform and a corresponding system (100) is depicted. As shown, a server (110) is provided to communicate with multiple computing devices (180), (182), (184), (186), (188), and (190) via a network connection (e.g., a computer network (105)). The server (110) is configured with a processing unit, such as a processor, that communicates with memory via a bus. The server (110) is shown with an AI platform (150) operatively coupled to a dialogue system (160) and a corresponding virtual agent (162), such as a chatbot, and a knowledge domain (170), such as a data source. A visual display (130), such as a computer screen or a smartphone, is provided to allow users to interface with a representation of the virtual agent (e.g., the chatbot (162)) on the display (130). Although Figure 1 The visual display is shown, but it should be understood that the display (130) may be replaced or supplemented by other interfaces (such as audio interfaces (e.g., microphone and speaker), audio-video interfaces, etc.).

[0032] The AI ​​platform (150) is operatively coupled to the network (105) to support interaction with a virtual dialogue agent (162) from one or more of the computing devices (180), (182), (184), (186), (188), and (190). More specifically, the computing devices (180), (182), (184), (186), and (188) communicate with each other and with other devices or components via one or more wired and / or wireless data communication links, wherein each communication link may include one or more of wires, routers, switches, transmitters, receivers, etc. In this networked arrangement, the server (110) and the network connection (105) enable communication detection, identification, and resolution. Other embodiments of the server (110) may be used with components, systems, subsystems, and / or devices other than those described herein.

[0033] The AI ​​platform (150) is shown herein as operatively coupled to a dialogue agent (162), which is configured to receive input (102) from various sources via a network (105). For example, the dialogue system (160) can receive input via the network (105) and utilize a data source (170) (also referred to herein as a knowledge domain or information corpus) to create output or response content.

[0034] As shown in the figure, the data source (170) is configured with multiple libraries. In this article, library (172) is shown as an example. A ), library B (172) B ), ... and Ku N (172) N Each library is populated with data in the form of feedback data and real data. In an exemplary embodiment, each library may point to a specific topic. For example, in an embodiment, the library... A (172) A (This can be filled with items that point to motion, library) B (172) B The library can be populated with items pointing to finance, etc. Similarly, in the embodiments, the library can be populated based on industry. The dialogue system (160) is operatively coupled to the knowledge domain (170) and the corresponding library.

[0035] The dialogue system (160) is an interactive AI interface used to support communication between a virtual agent and a non-virtual agent, such as a user, which can be a person or software and may be an AI virtual agent. The interactions that occur generate content called dialogue, which is stored in dialogue log files (also referred to as records). Each log file records interactions with the virtual dialogue agent (162). According to an exemplary embodiment, each dialogue log, such as a log file, is a record of questions presented to the dialogue system (160) and corresponding answers generated from the dialogue system (160). Therefore, the communication that occurs includes dialogues within an electronic platform between the user and the virtual agent.

[0036] The dialogue system (160) is operatively coupled to a knowledge base (140) to store records generated by the dialogue. As illustrated in the example, the knowledge base (140) is shown herein as a data structure having a representation of the dialogue. In this example, there are two data structures, DS A (142) A ) and DS B (142) B The number of data structures shown is for illustrative purposes and should not be considered restrictive. Each data structure is populated with a representation of the request, such as a question, and a corresponding generated response or response action, such as an answer. Figures 3-8 As shown and described, the dialogue questions and corresponding answers are represented by models, such as Abstract Meaning Representation (AMR) trees or parse trees. In an exemplary embodiment, the dialogue questions(s) are represented in one model, while the dialogue answers(s) are represented in another model. As an example, the first data structure DS A (142) A Using models to represent (multiple) problems Q,0 (142) Q,0 ) and models representing (multiple) corresponding answers A,0 (142) A,0 As shown in the diagram. Similarly, the second data structure DS B (142) B Using a model to represent the problem Q,1 (142) Q,1 ) and models representing the corresponding responses A,1 (142) A,1 The number of models shown and described herein is for illustrative purposes and should not be considered limiting. In practice, it can be expected that the knowledge base (140) may contain hundreds or thousands of dialogues and corresponding dialogue files or dialogue data structures, wherein each dialogue data structure has at least one request model and at least one response model.

[0037] Various computing devices (180), (182), (184), (186), (188), and (190) communicating with the network (105) may include access points to the dialogue system (162). In various embodiments, the network (105) may include local network connectivity and remote connectivity, enabling the AI ​​platform (150) to operate in environments of any size, including local and global, such as the Internet. Additionally, the AI ​​platform (150) serves as a backend system that makes various kinds of knowledge extracted from or represented in documents, web-accessible sources, and / or structured data sources available. In this way, processes populate the AI ​​platform (150), which also includes input interfaces for receiving requests and responding accordingly.

[0038] As shown in the figure, content can be represented in one or more models operatively coupled to the AI ​​platform (150) via a knowledge base (140). Content users can access the AI ​​platform (150) and the operatively coupled dialogue system (160) via a network connection to a network (105) or an Internet connection, and can submit natural language input to the dialogue system (160). The AI ​​platform (150) can then efficiently determine the output response related to the input by utilizing the operatively coupled data source (170) and tools including the AI ​​platform (150).

[0039] The AI ​​platform (150) is shown herein as having several tools to support the dialogue system (160) and the corresponding virtual agent (e.g., a chatbot) (162), and more specifically, tools aimed at improving the performance of the dialogue system (160) and the virtual agent (162). The AI ​​platform (150) employs several tools to interface with the virtual agent (162) and support improvements to its performance. These tools include a dialogue manager (152), an artificial intelligence (AI) manager (154), and a mentor (156).

[0040] The dialogue manager (152) interfaces with the dialogue system (160) via receiving natural language (NL) related to the interaction with the automated virtual dialogue agent (162). The received NL is shown herein as being populated and stored in a corresponding data structure in a knowledge base (140). Each dialogue event in the knowledge base (140) includes one or more requests and one or more corresponding responses or response actions.

[0041] The AI ​​manager (154) is shown herein as operatively coupled to the dialogue manager (152). The AI ​​manager (154) is used to apply received or acquired dialogue events to a learning procedure for knowledge gap assessment and remediation. As shown in the figure, the learning procedure (154) A) is operatively coupled to the AI ​​manager (154). The learning program (154) A The dialogue is explained by generating an explanation in the form of an explanatory request and an associated response. In an exemplary embodiment, the explanation is a rule between two or more concepts or a rule bridging two or more concepts. This explanation enables the learning program (154) to... A It can automatically identify the existence of knowledge gaps, if any. More specifically, the learning program (154) A The system determines whether the input (also referred to herein as an input instance) includes one or more concepts that are not present (e.g., not in) in the corresponding output instance or output action. The identification of knowledge gaps and instances requiring explanation is automatic. Examples of scenarios leading to knowledge gap identification include, but are not limited to, the following: identifying concepts in a question that are not presented as related to concepts in the corresponding true answer; identifying synonyms in the preferred answer to an item in a question; and identifying items that describe distinct contextual settings not explicitly mentioned in the question but present in the preferred answer. In an embodiment, the learning procedure (154) A This determines when the input instance and its corresponding output instance or (multiple) output actions are misaligned. In addition to identifying knowledge gaps, the learning program (154) A The AI ​​manager (154) dynamically requests knowledge fragments to expand the interpretation of (multiple) input instances and maps (multiple) requests to (multiple) output instances or (multiple) output actions, thereby effectively bridging the identified knowledge gaps. In an exemplary embodiment, the AI ​​manager (154) receives real data to guide the data source (170). Similarly, in an embodiment, the AI ​​manager (154) enriches the data source (170) with the interpreted and requested knowledge fragments. In an exemplary embodiment, the learning program (154) A This analyzes multiple pairs of input and output instances or actions, such as the input and output of a dialogue system. This extends the learning process (154). A The function is to identify one or more common features in two or more pairs of requests and responses, and then use those features to request knowledge fragments to reduce or eliminate knowledge gaps.

[0042] The AI ​​manager (154) represents the interpretation of the dialogue in the form of a model, such as, but not limited to, a model representing (multiple) questions. Q,0 (142) Q,0 ) and models representing (multiple) corresponding answers A,0 (142) A,0The model represents an AI manager (154) that can utilize the structure and functionality of the corresponding model to compare content (e.g., text of question-answer pairs) and determine any similarities or differences corresponding to conceptual relationships. In an exemplary embodiment, the model is a representation of dialogue events in a subtree format. The AI ​​manager (154) quantifies knowledge gaps by measuring the distance between subgraphs with common node labels. In an exemplary embodiment, the measured distance corresponds to the complexity of the identified knowledge gap or an indicator of the complexity of the identified knowledge gap. In addition to comparison, the AI ​​manager (154) can generate one or more questions for the dialogue manager (152) to communicate through the dialogue system (160) and the corresponding virtual agent (162). In an exemplary embodiment, the dialogue manager (152) receives one or more answers to the generated questions. The received answers serve as indicators of the identified knowledge gaps and are transmitted to the AI ​​manager for knowledge gap assessment and remediation.

[0043] This illustrates the operation of the mentor (156) coupled to the dialogue manager (152), AI manager (154), and dialogue system (160). The mentor (156) optimizes the virtual agent (162) with respect to the corresponding dialogue events. This optimization takes the form of leveraging the chatbot platform to facilitate and enable system interaction to gather domain-specific relationships from correct responses, such as truth values ​​or positive feedback. Details of the optimization are... Figure 3 The diagram shows and describes how optimizing bridging or mitigating knowledge gaps improves the performance of the dialogue system (160) by enhancing the accuracy of responses related to requests.

[0044] Dialogue events created or enabled by the dialogue system (160) can be handled by IBM Watson. ®The server (110) and the corresponding artificial intelligence platform (150) process the dialogue. The dialogue manager (152) performs analysis of the received natural language using various inference algorithms. Hundreds or even thousands of inference algorithms can be applied, each performing different analyses, such as comparisons. For example, some inference algorithms may focus on matching items and synonyms within the language of the received dialogue with corresponding responses or response actions. In one embodiment, the dialogue manager (154) may process electronic communications to identify and extract features within the communications. Whether by using feature extraction and feature representation or by using an alternative platform for processing electronic records, the dialogue manager (152) processes dialogue events to attempt to identify and parse the events and behavioral characteristics of the dialogue events. In an exemplary embodiment, behavioral characteristics include, but are not limited to, language and knowledge. In one embodiment, the platform identifies grammatical components in requests and corresponding responses or response actions, such as nouns, verbs, adjectives, punctuation marks, etc. Similarly, in one embodiment, one or more inference algorithms may examine temporal or spatial features in the language of electronic records.

[0045] In some illustrative embodiments, the server (110) may be IBM Watson obtained from International Business Machines Corporation in Armonk, New York. ® The system is enhanced using the mechanisms of the illustrative embodiments described below.

[0046] The dialogue manager (152), AI manager (154), and instructor (156), collectively referred to below as AI tools, are shown as embodied in or integrated into an artificial intelligence platform (150) on the server (110). The AI ​​tools may be implemented in a separate computing system (e.g., 190) connected to the server (110) across a network (105). Regardless of their embodiment, the AI ​​tools are used to evaluate dialogue events, extract behavioral characteristics from requests and responses, selectively identify knowledge gaps, and dynamically request inputs through proactive interpretation, bridging knowledge gaps and improving question transformation and knowledge representation.

[0047] In the selected example embodiment, the dialogue manager (152) can be configured to apply NL processing to identify behavioral characteristics of dialogue events. For example, the NL manager (152) can perform sentence structure analysis, which requires parsing of the main statement and parsing of grammatical terms and parts of speech. In one embodiment, the NL manager (152) can use a slotted syntax logic (SGL) parser to perform the parsing. The NL manager (152) can also be configured to apply one or more learning methods to match detected content with known content to determine behavioral characteristics and assign values ​​to the behavioral characteristics.

[0048] The range of information processing systems that can use an artificial intelligence platform (150) ranges from small handheld devices such as handheld computers / mobile phones (180) to mainframe systems such as mainframe computers (182). Examples of handheld computers (180) include personal digital assistants (PDAs), personal entertainment devices such as MP4 players, portable televisions, and compact disc players. Other examples of information processing systems include pen or tablet computers (184), laptop or notebook computers (186), personal computer systems (188), and servers (190). As shown, the various information processing systems can be networked together using a computer network (105). The types of computer networks (105) that can be used to interconnect the various information processing systems include local area networks (LANs), wireless local area networks (WLANs), the Internet, the public switched telephone network (PSTN), other wireless networks, and any other network topologies that can be used to interconnect the information processing systems. Many information processing systems include non-volatile data storage, such as hard disk drives and / or non-volatile memory. Some information processing systems may use separate non-volatile data storage (e.g., the server (190) uses non-volatile data storage (190) A ), and mainframe computers (182) use non-volatile data storage (182) A Non-volatile data storage (182) A () can be a component outside of each information processing system, or it can be a component inside one of the information processing systems.

[0049] Information processing systems used to support artificial intelligence platforms (150) can take many forms, some of which are... Figure 1 As shown in the diagram. For example, an information processing system can take the form of a desktop computer, server, portable computer, laptop computer, notebook computer, or other form factor computer or data processing system. Furthermore, an information processing system can take other form factors, such as a personal digital assistant (PDA), gaming device, ATM machine, portable telephone device, communication device, or other device including a processor and memory.

[0050] Application Programming Interface (API) is understood in this field as a software intermediary between two or more applications. About Figure 1 The artificial intelligence platform (150) shown and described herein includes one or more APIs that can be used to support one or more of the tools (152), (154), and (156) and their associated functions. References Figure 2A block diagram (200) is provided illustrating tools (152), (154), and (156) and their associated APIs. As shown, multiple tools are embedded within an AI platform (205), including a dialogue manager (252) associated with API0 (212), an AI manager (254) associated with API1 (222), and a mentor (256) associated with API2 (232). Each API can be implemented in one or more languages ​​and interface specifications. API0 (212) provides functional support to receive and evaluate dialogue events and determine behavioral characteristics; API1 (222) provides functional support to apply the received dialogue events to a learning process to identify and address knowledge gaps; and API2 (232) provides functional support to further optimize the virtual dialogue using solutions to knowledge gaps. As shown, each of APIs (212), (222), and (232) is operatively coupled to an API coordinator (260), or coordination layer, which is understood in the art as serving as an abstraction layer that transparently strings the individual APIs together. In the embodiments, the functionality of individual APIs may be combined or integrated. Thus, the configuration of the APIs shown herein should not be considered limiting. Therefore, as illustrated herein, the functionality of a tool may be embodied or supported by its respective API.

[0051] refer to Figure 3A flowchart (300) is provided illustrating the process of enriching a corpus of specific semantic relations for a trusted domain. A virtual communication platform, referred to herein as a chatbot, serves as the basis or platform for corpus enrichment, as shown and described. A question is presented or submitted in the chatbot platform (302) to request or initiate chatbot interaction and response data (304). The chatbot employs natural language processing (NLP) and a knowledge corpus to generate one or more answers to the submitted question (306). In embodiments with multiple answer generation, the generated answers are referred to as alternative answers. It should be understood in the art that the generated responses may not provide accurate or expected answers. If the user is dissatisfied with the answer and wants to continue the interaction, the system may collect implicit negative feedback (308) and then return to step (304) for continued chatbot interaction. In embodiments, continued interaction provides implicit negative feedback. If the user wants to terminate the interaction, the process continues with the collection of feedback (310), which in embodiments includes the collection of questions and corresponding responses (310) in response to a final feedback request from the user (312). The final feedback can be positive or negative and corresponds to the final answer provided by the chatbot platform. Examples of feedback include "The answer is good" or "The answer is helpful" as positive feedback, and "The answer is bad" or "The answer is not helpful" as examples of negative feedback. In an exemplary embodiment, in the case of negative feedback, multiple-choice questions may be used to limit the scope of errors, such as "Incorrect content" or "Content not found".

[0052] At (314), the feedback (e.g., final feedback) is stored in a repository, and at step (316), the system begins to utilize this feedback to enrich domain knowledge. This enrichment can respond to either positive or negative feedback. Similarly, this enrichment can be achieved through a chatbot platform or through truth values ​​provided by subject matter experts (SMEs). Similarly, if a knowledge gap exists, the enrichment can assess the existence of the knowledge gap and address such gaps through domain knowledge enrichment via feedback interaction through the chatbot (162). It should be understood in the art that various methods may have been used to generate responses to the chatbot, and therefore, some matches between questions (e.g., inputs) and corresponding generated answers may be partial, such as partially good or partially bad. Although they may be considered good answers overall, there may be knowledge gaps from which additional information can be benefited. Similarly, when the feedback is negative, the system may consider the answer to be good and therefore needs or can benefit from identifying information or knowledge to clarify the discrepancies.

[0053] Learning, also referred to herein as learning interaction, is shown here as being divided into feedback interaction (316) and knowledge enrichment interaction (318). Feedback interaction utilizes the chatbot platform to facilitate and enable system interaction to collect domain-specific relations, such as truth values ​​or positive feedback, from correct responses. These relations are used to enrich domain knowledge. Feedback interaction (316) is followed by the generation of interpretive prompts (320), the details of which include knowledge gap assessment, such as... Figure 5 As shown and described. An explanation request is generated and presented to the chatbot platform for learning interaction (322). Feedback is generated by the chatbot platform in the form of an explanation response, and this feedback undergoes explanation processing (324). Figure 6 The details of the interpretation process are shown and described in detail. Knowledge enrichment is performed asynchronously and is requested from the subject matter expert (SME), as shown in step (318). Details of knowledge enrichment are provided in... Figure 4 The information is shown and described in section (318). The knowledge enrichment at step (318) utilizes an interaction channel with one or more SMEs, which the system uses to extract or otherwise obtain questions and corresponding answers, and to request explanations in the form of truth values. In an exemplary embodiment, truth values ​​are used to augment domain knowledge to enhance the accuracy of data communication through the chatbot platform. Truth values ​​can be positive, such as a correct answer, or negative, such as an invalid answer. In an embodiment, truth values ​​are collected in a dedicated question-and-answer data structure, items are extracted from this data structure, and this data structure is used to generate requests for explanations from one or more SMEs. Thus, the proactive explanation channel is used for domain knowledge augmentation, also referred to herein as corpus enrichment, to enhance and expand the capture of concepts and relationships for future interactions.

[0054] refer to Figure 4 A flowchart (400) is provided to illustrate the process for knowledge enrichment interactions. As shown and described, an explanation request is generated, where corresponding patterns of knowledge are used to develop questions to enrich domain knowledge with one or more relationships. The generation of explanation options begins with the results of a knowledge gap assessment. Figure 5As shown and described, the first and second data structures are filled with matching and non-matching concepts and relations, which are used here as inputs (402). The scope of knowledge richness is based on learning with explicit feedback, which can be positive or negative feedback and may be combined with positive or negative implicit feedback. During chatbot platform training, one or more subject matter experts (SMEs) generate or produce truth values ​​in the form of a set of questions and corresponding answers, from which the chatbot platform learns how to answer presented questions and respond, such as in chatbot training. Truth values ​​can be positive, such as a good answer to a presented question, or negative, such as an invalid answer to a presented question. In an exemplary embodiment, truth values ​​are collected in corresponding data structures. As shown, items are extracted from the corresponding questions and answers, whether matching or not, and an explanation request (404) is generated targeting the SME. The explanation request can be individual-based or in one or more bundles, where questions and corresponding answers are grouped together, such as in a CSV file. The SME generates an explanation response (406), which is transmitted via a communication channel for explanation processing, for the analysis and collection of new semantic relations (408), such as Figure 7 As shown and described. Furthermore, SME generates truth values ​​(410) that are filled into the corresponding data structure.

[0055] refer to Figure 5 A flowchart (500) illustrating the generation process for interpretation is provided. As shown and described, the initial aspect of the interpretation generation process points to knowledge gap assessment to process natural language items based on questions and answers obtained from a chatbot environment, where these items are represented in a model for machine understanding and consumption. The process begins with input (502) in the form of question text and corresponding answer text, which in this embodiment is obtained from a chatbot platform. Regarding the question, concepts and correspondences within the question are extracted using natural language tools (504). In an exemplary embodiment, the tool analyzes the question to identify its components, such as objects, verb-object relationships, or noun-object relationships. Information in the corresponding knowledge domain is used to identify and extract concepts, such as fault actions, attributes, product management actions, product names, product ingredients, etc. Similarly, in this embodiment, a classifier based on a neural language model or sequence model may be employed to label which items are relevant to the corresponding knowledge domain.

[0056] The representations of the components of the problem identified in step (504) (also referred to herein as requests) are stored in a corresponding data structure (506). For example, in an embodiment, the identified problem components are represented as an Abstract Meaning Representation (AMR) tree or parse tree. An AMR is a semantic representation that expresses the logical meaning of a sentence having a rooted, directed, acyclic graph. An AMR associates semantic concepts with nodes on the graph, and the relationships are labeled edges between concept nodes. In an exemplary embodiment, an AMR represents the semantic representation of a sentence in a hierarchical structure, which is an organization technique in which items are layered or grouped to reduce complexity.

[0057] Following the extraction and representation in steps (504) and (506) respectively, the answers to the question are analyzed (506) in a manner similar to question analysis. More specifically, at step (508), one or more answers to the question are generated and subjected to analysis. In an exemplary embodiment, the answers are obtained from a corresponding knowledge base or knowledge domain. The analysis subjectes the answers to NL processing similar to that of the question, where one or more components of the question (e.g., subject or object, and corresponding characteristics, etc.) are identified. The analysis identifies the relevance of the extracted concept and relation instances, and in an embodiment, the concept and relation instances are ranked and represented or characterized in a data structure representing the question. In an exemplary embodiment, information in the knowledge domain is used for analysis and identification. This information includes, but is not limited to, specific types of concepts or relations, such as fault actions and attributes, product management actions (e.g., restart, deletion, and configuration), product names, and product components. Similarly, in an embodiment, a classifier based on a neural language model or sequence model is used to label which items are relevant to the domain. In an exemplary embodiment, the extracted concepts and relations may be arranged in a hierarchy based on ranking. Similar to question processing, concepts within the answer text and relationships from the questions relating to the answer are extracted, and representations of the components of the answer identified in step (510) (also referred to herein as responses) are stored in corresponding data structures (512). Similar to question processing, in this embodiment, the identified answer components are represented as an Abstract Meaning Representation (AMR) tree or a parse tree. In an exemplary embodiment, the representations of the question and answer may include multiple data structures representing multiple cognitive features. Therefore, both the question and the corresponding answer text undergo concept and relationship extraction.

[0058] The list of elements of questions and answers populated in the corresponding data structure or model at steps (504) and (508) corresponds to specific selection criteria, such as matching or being different. The list can be generated by applying a comparison method specific to each type of feature representation. For example, two data structures can be received as input along with the output from a comparison method that generates a list of elements present in both data structures, a list of elements present only in the first data structure, and a list of elements present only in the second data structure. For a relational graph, differences in semantic or syntactic relational graph subtrees are identified to determine the same subtrees or subtrees within a distance threshold. For example, the distance can be measured between subgraphs with common node labels, such as the number of relationships initiated or ended at the common node.

[0059] After step (512), the representations of the questions and corresponding answers are subjected to comparison to extract overlaps and differences (514). The comparison determines differences in semantic or syntactic relational graph subtrees to identify those subtrees that are the same or within a distance threshold. The distance can be measured between subgraphs with common node labels, such as the number of relationships that start or end at the common node, which are different, i.e., have different nodes in the triple (common node, common relationship, node), or are missing, i.e., the common node and the relationship in only one of the compared trees. For example, in an exemplary embodiment, the question in a scenario is "The product battery must be replaced" and is shown with the following AMR representation:

[0060]

[0061] Based on this example, the subtree representation has a common node "replace - 01", and the distance of the relevant subtree is 1 because the relationships <replace, "action", battery> and <batt, action, unknown> are different. The analysis determines that "battery" is a concept of the "component" type, while "batt" is unknown. Thus, the comparison is performed using the data structures created at steps (506) and (512).

[0062] Then it is determined whether the extracted question concepts and relationships as a whole are represented in the extracted answer concepts and relationships (516). A positive response to this determination is an indication that the question and answer match (e.g., align) and there is no knowledge gap, and the comparison process terminates (518). In an exemplary embodiment, the evaluation of step (516) is directed to concepts represented in layers of hierarchy. A negative response to the determination at step (516) is followed by a subsequent determination to evaluate concepts and relationships relative to the relevance identified in the question at step (506) and their representation in the answer (524). In an exemplary embodiment, the evaluation of step (524) is directed to different layers in hierarchy, such as concepts different from those evaluated in step (516). A positive response to the evaluation at step (524) is an indication of the evaluated question or aspects of the question identified as existing within the corresponding answer or, in one embodiment, within the evaluated or identified concepts of the corresponding answer. Thus, as shown herein, the overlap between the answer and the question can be directed to the evaluation of one or more concepts identified therein.

[0063] As shown and described, the evaluation at step (524) targets concepts identified or represented within the question and may not be entirely relevant to the question. Following a positive response to the evaluation at step (524), it is determined whether the evaluation between the question and answer should continue (526) by evaluating additional concepts represented within the question's hierarchy. In an exemplary embodiment, the user may refer to an element of a concept represented in the hierarchy, but the relevant content pertains to related concepts at different levels within the hierarchy. Continuing the evaluation corresponding to a positive response at step (526) would require additional time and, in embodiments, could be disruptive to the chatbot experience. A negative response at step (526) terminates the evaluation process as shown here by jumping to the determining sequence that begins at step (518). However, following either a positive response at step (526) or a negative response at step (524), the question text is transformed (528) based on equivalents of other relationships available in domain knowledge, such as is-a, symptom-action, and others. The transformation at step (528) aims to identify the differences between the question and the corresponding answer. In an exemplary embodiment, one or more language models are used to determine the equivalent sentence of the question that matches any item found in any previous match corresponding to the question. It is then determined whether any new alternative expressions of the question have been generated (530). A positive response in step (530) is followed by a return to step (514), and a negative response is followed by a jump to the termination sequence that begins at step (518).

[0064] At the end of the evaluation, this is shown as a negative response to the determinations at steps (526) and (530), and after step (518), the corresponding first and second data structures are populated. More specifically, representations of matching concepts, relations, and graphical regions are populated in the first corresponding data structure (520). Similarly, representations of non-matching concepts, relations, and graphical regions are populated in the second corresponding data structure (522). Although shown sequentially, in an exemplary embodiment, the population of the first and second data structures in steps (520) and (522) may occur in parallel or in an alternative order. The first data structure is populated with question and answer elements corresponding to a specific selection criterion, e.g., matching in both representations, and the second data structure is populated with question and answer elements corresponding to a specific selection criterion, which are distinct, e.g., existing only in one of the representations. A list is generated by applying a comparison method specific to each type of feature representation. For example, in a concept list, the process takes two lists as input, iterates through and compares the items in the lists, and outputs a list of elements present in both lists, such as a matching concept, and lists of elements present only in the first input and lists of elements present only in the second input, such as a non-matching concept. In an embodiment utilizing AMR, the process may output subgraph pairs with the same root node label, and edges in the graph with the same label and the same neighboring node labels, such as a matching concept.

[0065] As shown and described, an assessment is performed to determine or identify the existence of a knowledge gap, which is analogous to determining that there is no match between a question and an answer. In an exemplary embodiment, the term matching is used for both exact matching and matching of synonyms. The use of the term "match" is based on a comparison of the words expressed in the question and the presence of those words in the answer. A knowledge gap can take the form of a missing connection, such as a missing synonym or a missing relation inference. Although the system provides feedback in the form of final feedback from the user, the system analyzes the question and the corresponding answer stored in a repository. In an exemplary embodiment, a user-oriented engagement strategy is provided relative to the knowledge gap assessment to address users with different skill levels and different patient levels, and thus the engagement strategy selectively guides the learning interaction for knowledge gaps. A partial match may be a satisfactory or good response, and the system can learn what makes a good match and can learn additional relations that bring confidence to the answer. Therefore, the engagement strategy can learn from both the collected positive and negative feedback.

[0066] From Figure 5 The output of the knowledge gap assessment shown generates one or more explanatory options regarding the relationship between questions and answers. For example... Figure 4 and Figure 5As shown, the initial aspect points to the automatic assessment of knowledge gaps, where knowledge gaps or missing knowledge gaps are identified. Once a knowledge gap is identified, the process proceeds to options on how to fill it. Artifacts of the interpretation management system provided for several types of knowledge are shown here as rules for mismatches with interpretation mapping artifacts, presentation templates for generating interpretation outputs for users, and management strategies to be applied when more than one interpretation type matches. The following table, Table 1, provides examples of artifacts of the interpretation management system provided for several types of knowledge:

[0067]

[0068] Table 1

[0069] The following table, Table 2, is an example of a presentation template for the interpretation of the output generated by the user:

[0070]

[0071] Table 2

[0072] The following table, Table 3, provides examples of management strategies to be applied when more than one interpretation type matches:

[0073]

[0074] Table 3

[0075] refer to Figure 6 A flowchart (600) is provided to illustrate the process of generating explanatory options as semantic relationships between question and answer phrases to enrich domain knowledge. It can be understood that certain aspects of the semantic relationships between questions and answers may have been previously identified and represented or stored in domain knowledge. For example, in... Figure 5 In this context, existing semantic relationships may already be used to identify matches between questions and answers. In an exemplary embodiment, both the differences and similarities between questions and answers are analyzed, and the similarity analysis enriches the domain knowledge. The process of generating explanation options uses patterns from the corresponding domain knowledge to develop questions in order to enrich the domain with relationships. Domain knowledge provides information about what is already known based on equivalences and relationships.

[0076] As shown in the figure, Figure 5The input is provided in the form of a data structure containing matching and non-matching knowledge items generated in the process (602). It is understood that the knowledge items from the input may contain positive or negative feedback. For positive feedback, if multiple differences exist, they are ordered according to complexity, such as the distance between adjacent graphs, and in the embodiment, starting with smaller distances. Differences are filtered based on the user's expected response to the explanation. For example, a user with lower skills would not be required to explain differences represented by larger distances between graphs or subgraphs with common node labels. The choice of explanation depends on the type of relations captured in the knowledge domain. For negative feedback, the selection can be made according to different criteria. The artifacts of the management system provide the following types of knowledge: rules that associate a set of patterns with explanation types, a presentation template for generating outputs for users to provide explanations, and a system or management strategy to be applied when more than one explanation type matches the input. For each pattern in the rules that matches the input, an explanation instance is generated as output (604). The output from step (604) is a list of pattern matching instances. In an exemplary embodiment, a pattern may be applied to an additional graphical area. Following step (604), a system or management policy is applied to the list of explanation instances to generate output in the form of a final explanation instance to be presented to the user (606). In an exemplary embodiment, a set of explanation types is selected for presentation in step (606). A request for explanation is created (608) by applying one or more presentation template patterns to create output to be sent to the chatbot platform. Details of the call are recorded (610), where patterns are matched within the explanation management system for processing. Presentation templates are organized based on interaction, explanation type, explanation relationship, or output portions (e.g., open and close statements).

[0077] Here are examples of structured questions for users interacting online:

[0078]

[0079] The following are example annotation issues used for batch interaction:

[0080]

[0081] Therefore, as shown in this paper, a set of presentation templates in question and multiple-choice format is used to generate output for user-provided explanations. The questions limit how you represent relationships in the corresponding knowledge graph.

[0082] As shown and described herein, the strategy corresponds to wording one or more questions (e.g., prompts) to generate a synchronous response that explains the semantic relationship or lack thereof between the question and the answer. Similarly, in embodiments, an asynchronous response can be obtained by using an explanation of the requested answer or a batch of answers from the SME. Thus, explanations can be obtained either through the use of a synchronous channel of the chatbot platform or through an asynchronous channel of the SME.

[0083] refer to Figure 7 A flowchart (700) is provided to illustrate the processing and recording explanation of the relationship between questions and answers. The explanation of processing and recording utilizes... Figure 6 The interpretation context determined in the interpretation context is processed for the interpretation response of an interpretation instance (702). In an exemplary embodiment, each interpretation instance has a corresponding interpretation identifier. Each interpretation identifier is a reference to metadata stored in an interpretation management system that collects all details about the interpretation instance. The collected details include one or more of the following: question, answer, concept and relation similarities and differences, and interpretation goals and parameters. Using the interpretation identifier or other details in the interpretation response, an interpretation context including details about the interpretation goals and parameters is obtained (704). Thereafter, using the interpretation selection selected in the interpretation response and interpretation metadata, the type of relation identified by the user is determined (706). It is then determined whether the user selected "I don't know" as the response (708). After a positive response in step (708), the interpretation is saved for review by the SME (710), and after a negative response in step (708), it is determined whether the user selected "irrelevant concept" as the response (712). After a positive response determined in step (712), it is determined whether there is another interpretation request for the current context (714). After a positive response is determined at step (714), the process returns to step (706), and a negative response ends the interpretation evaluation.

[0084] The negative response determined at step (712) is an indication that a knowledge artifact can be created from the relation indicated by the user. A mapping is created to associate the interpretation type (e.g., relation type and concept type) with one or more domain knowledge components and artifact types (714), followed by an update action (716) on the knowledge domain of each knowledge indication that can be generated from the relation indicated by the user.

[0085] refer to Figure 8A diagram (800) is provided to illustrate an example relation-knowledge artifact-mapping. As shown, the map has two columns, including a relation-concept type column (810) and a domain knowledge component-artifact column (820). In this example, equivalence relations (812) of verbs, nouns, and phrases, such as equivalent meanings expressed in verb, noun, or phrase format, are integrated as synonyms in the search engine (822). Similarly, "is-a" or hypernym relations (814) are integrated in the search engine extended synonyms (824), causal or negation relations (816) are integrated as knowledge graph or search engine extended synonyms (826), and relation synonym remedies (818) are knowledge graph artifacts (828). For each knowledge artifact that can be generated, an update action (718) for the domain knowledge is initiated. The update action in step (718) can be applied immediately or collected and batched in a predefined system maintenance window. Thus, as shown in this paper, the knowledge gap undergoes analysis to address its interpretation.

[0086] return Figure 3 Furthermore, as shown and described, active learning is achieved based on the interpretation of semantic relationships. As shown, interpretation requests (318) are assigned to the chatbot platform for learning interactions (322) and soliciting truth values ​​via one or more SMEs (318). In an exemplary embodiment, when the correspondence between a question and an answer is weak, for example, not strong, the system generates one or more candidate interpretability relationships that can enforce the correspondence, for example, as Figure 5 As shown and described, and utilizing interfaces such as chatbots, the relevance is defined. The learning interaction allows or supports online evaluation of the explanation. In an exemplary embodiment, feedback on the quality of the response can be used to trigger learning from the explanation, such as implicit or explicit feedback, positive or negative feedback, etc. Similarly, in embodiments, a strategy can be used to determine when to trigger an interaction for the explanation request at step (318) in order to control the impact on user satisfaction. An example strategy could be in the form of implementing only negative feedback, or another example strategy could be the user's skill level, such as a novice. As shown, the learning interaction (320) generates an explanation response (322), which is processed and records an explanation of the relationship between the question and the answer (324), such as Figure 7 As shown and described, after steps (324) and (318), the processing of interpreting the relationship between the response and the identification is stored in the domain knowledge (326). Therefore, the relationship between the target phrase and any missing information, and the corresponding phenomena, are identified and stored in the knowledge base.

[0087] like Figures 1-8The computer system, program product, and method shown and described are provided to compare question and answer pairs, particularly the text therein, and determine similarities and differences in conceptual relationships regarding how the answers involve or reference expected relationships derived from the questions. The similarities and differences are used to generate specific multiple-choice questions about possible causes of the differences. The questions are presented in an AI interaction platform (e.g., a chatbot platform) to extract relationships from correct responses (e.g., truth values ​​or positive feedback). The extracted relationships, as well as new relationships in the embodiments, such as explanatory information, are identified and used to augment domain knowledge, thereby expanding domain knowledge with the captured concepts and relationships.

[0088] The embodiments shown and described herein can take the form of a computer system used with an intelligent computing platform for enriching domain knowledge. Aspects of tools (152), (154), and (156) and their associated functionality can be embodied in a single-location computer system / server, or, in the embodiments, configured in a cloud-based system with shared computing resources. References Figure 9 A block diagram (900) is provided illustrating an example of a computer system / server (902), which, hereinafter referred to as such, communicates with a cloud-based support system (910) to achieve the above. Figures 1-8 The host (902) is the system, tool, and process described herein. The host (902) may operate with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with the host (902) include, but are not limited to, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and file systems (e.g., distributed storage environments and distributed cloud computing environments) that include any of the aforementioned systems, devices, and their equivalents.

[0089] The host (902) can be described in the general context of computer system executable instructions such as program modules that are executed by the computer system. Typically, a program module may include routines, programs, objects, components, logic, data structures, etc., that perform a specific task or implement a specific abstract data type. The host (902) can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can reside in local and remote computer system storage media, including memory storage devices.

[0090] like Figure 9As shown, the host (902) is illustrated as a general-purpose computing device. Components of the host (902) may include, but are not limited to, one or more processors or processing units (904) (e.g., hardware processors), system memory (906), and buses (908) that couple various system components, including system memory (906), to the processors (904). The bus (908) represents one or more of several types of bus architectures, including memory buses or memory controllers, peripheral buses, accelerated graphics ports, and processor or local buses using various bus architectures. By way of example and not limitation, these architectures include Industry Standard Architecture (ISA) buses, Microchannel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses. The host (902) typically includes various computer system readable media. This media can be any available media accessible to the host (902), and it includes volatile and non-volatile media, removable and non-removable media.

[0091] The system memory (906) may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) (930) and / or cache memory (932). By way of example only, the storage system (934) may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, and generally referred to as a "hard disk drive"). Although not shown, a disk drive may be provided for reading from and writing to a removable, non-volatile disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM, or other optical media. In this case, each may be connected to a bus (908) via one or more data media interfaces.

[0092] A program / utility (940) having a set (at least one) of program modules (942), along with an operating system, one or more applications, other program modules, and program data, may be stored in system memory (906), as an example and not a limitation. Each of the operating system, one or more applications, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. Program modules (942) typically perform the functions and / or methods of an embodiment to dynamically interpret and understand request and action descriptions and effectively add relevant domain knowledge. For example, the set of program modules (942) may include, for example, Figure 1 The tools shown are (152), (154) and (156).

[0093] The host (902) can also communicate with one or more external devices (914), such as a keyboard, indicating device, etc.; a display (924); one or more devices that enable a user to interact with the host (902); and / or any device that enables the host (902) to communicate with one or more other computing devices (e.g., a network card, modem, etc.). Such communication can occur via an input / output (I / O) interface (922). Furthermore, the host (902) can communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet), via a network adapter (920). As described, the network adapter 920 communicates with other components of the host 902 via a bus 908. In one embodiment, multiple nodes of a distributed file system (not shown) communicate with the host (902) via the I / O interface (922) or via the network adapter (920). It should be understood that, although not shown, other hardware and / or software components can be used in conjunction with the host (902). Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0094] In this document, the terms “computer program medium,” “computer-usable medium,” and “computer-readable medium” are used to generally refer to media such as main memory (906), including RAM (930), cache (932), and storage systems (934), such as removable storage drives and hard disks installed in hard disk drives.

[0095] The computer program (also known as computer control logic) is stored in memory (906). The computer program can also be received via a communication interface, such as a network adapter (920). When run, such a computer program enables the computer system to perform the features of this embodiment as discussed herein. In particular, when run, the computer program enables the processing unit (904) to perform the features of the computer system. Thus, such a computer program represents the controller of the computer system.

[0096] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, dynamic or static random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), magnetic storage devices, portable optical disc read-only memory (CD-ROM), digital multifunction discs (DVDs), memory sticks, floppy disks, mechanical encoding devices such as punch cards or raised structures in recesses on which instructions are recorded, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0097] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0098] Computer-readable program instructions used to perform the operations of this embodiment may be assembly instructions, instruction set architecture (ISA) instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server cluster. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of the embodiment by utilizing state information from the computer-readable program instructions.

[0099] The functional tools described in this specification are labeled as managers. Managers can be implemented in programmable hardware devices such as field-programmable gate arrays, programmable array logic, programmable logic devices, etc. Managers can also be implemented as software for processing by various types of processors. The executable code identifying a manager may include, for example, one or more physical or logical blocks of computer instructions, which may be organized, for example, as objects, procedures, functions, or other constructs. However, the executable code of the identified manager does not need to be physically located together, but may include different instructions stored in different locations that, when logically combined, include the manager and achieve the manager's stated purpose.

[0100] In practice, the manager of executable code can be a single instruction or multiple instructions, and can even be distributed across several different code segments, different applications, and across several memory devices. Similarly, operational data can be identified and displayed within the manager, and can be represented in any suitable form and organized within any suitable type of data structure. Operational data can be collected as a single dataset, or it can be distributed across different locations including different storage devices, and can exist at least partially as electronic signals on a system or network.

[0101] Now for reference Figure 10An illustrative cloud computing network (1000) is shown. As illustrated, the cloud computing network (1000) includes a cloud computing environment (1050) with one or more cloud computing nodes (1010), whose local computing devices used by cloud consumers can communicate with the cloud computing nodes. Examples of such local computing devices include, but are not limited to, personal digital assistants (PDAs) or cellular phones (1054A), desktop computers (1054B), laptop computers (1054C), and / or automotive computer systems (1054N). The individual nodes within the nodes (1010) can also communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment (1000) to provide infrastructure, platform, and / or software as a service, without requiring cloud consumers to maintain resources on their local computing devices. It should be understood that... Figure 10 The types of computing devices (1054A-N) shown are for illustrative purposes only, and the cloud computing environment (1050) can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0102] Now for reference Figure 11 This shows the result of Figure 10 The cloud computing network provides a set of functional abstraction layers (1100). It should be understood beforehand that... Figure 11 The components, layers, and functions shown are for illustrative purposes only, and the embodiments are not limited thereto. As depicted, the following layers and corresponding functions are provided: hardware and software layer (1110), virtualization layer (1120), management layer (1130), and workload layer (1140).

[0103] The hardware and software layer (1110) includes hardware and software components. Examples of hardware components include mainframes, in one example being IBM. ® zSeries ® System; based on a RISC (Reduced Instruction Set Computer) architecture server, in an example IBM pSeries ® In the system; IBM xSeries ® System; IBM BladeCenter ® Systems; storage devices; networking and interconnection components. Examples of software components include web application server software, such as IBM WebSphere in one example. ® Application server software; and database software, in one example being IBM DB2. ®Database software. (IBM, zSeries, pSeries, xSeries, BladeCerter, WebSphere, and DB2 are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide.)

[0104] The virtualization layer (1120) provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers; virtual storage; virtual networks, including virtual private networks; virtual applications and operating systems; and virtual clients.

[0105] In the example, the management layer (1130) can provide the following functions: resource provisioning, metering and pricing, user portal, service level management, and SLA planning and enforcement. Resource provisioning provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and pricing provides cost tracking when utilizing resources in the cloud computing environment, as well as invoicing or issuing invoices for consuming these resources. In the example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. The user portal provides consumers and system administrators with access to the cloud computing environment. Service level management provides the allocation and management of cloud computing resources to meet required service levels. Service level agreement (SLA) planning and enforcement provides the pre-scheduling and procurement of cloud computing resources, where future demand is anticipated based on the SLA.

[0106] The workload layer (1140) provides examples of functionalities that can be leveraged in a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include, but are not limited to: map creation and navigation; software development and lifecycle management; virtual classroom education delivery; data analytics and processing; transaction processing; and natural language enrichment.

[0107] Although specific embodiments of the invention have been shown and described, it will be apparent to those skilled in the art that changes and modifications can be made based on the teachings herein without departing from the invention and its broader aspects. Therefore, the appended claims are intended to cover within their scope all such changes and modifications that fall within the true spirit and scope of the embodiments. Furthermore, it should be understood that the embodiments are defined solely by the appended claims. Those skilled in the art will understand that if a specific number of claim elements is intentional, such intention will be explicitly stated in the claims, and without such a statement, there is no such limitation. For non-limiting examples, to aid understanding, the appended claims include the use of the introductory phrases “at least one” and “one or more” to introduce claim elements. However, the use of such phrases should not be construed as implying that introducing a claim element by the indefinite article “a” or “an” limits any particular claim containing such an introduced claim element to an embodiment containing only one such element, even when the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” or “an”; the same applies to the use of definite articles in the claims. As used herein, the term “and / or” means one or both of them (or one or any combination or all of the terms or expressions mentioned).

[0108] Embodiments of the present invention may be systems, methods, and / or computer program products. Furthermore, selected aspects of this embodiment may take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments combining software and / or hardware aspects, all of which may be collectively referred to herein as “circuit,” “module,” or “system.” Additionally, aspects of embodiments of the present invention may take the form of computer program products implemented in computer-readable storage media (or media) having computer-readable program instructions for causing a processor to execute aspects of embodiments of the present invention. The disclosed systems, methods, and / or computer program products thus implemented are operable to support natural language richness.

[0109] This document describes aspects of the present embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to the embodiments. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0110] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other devices to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0111] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions mentioned in the blocks may occur in a non-linear order as shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0113] It should be understood that although specific embodiments have been described herein for illustrative purposes, various modifications may be made without departing from the spirit and scope of the embodiments. Therefore, the scope of protection of the embodiments is limited only by the appended claims and their equivalents.

Claims

1. A computer system, comprising: The processor is operatively coupled to the memory; An artificial intelligence (AI) platform, communicating with the processor, the AI ​​platform having tools to guide the performance of the dialogue system, the tools including: The dialogue manager receives natural language (NL) related to interactions with the automated virtual dialogue agent of the dialogue system, the NL including one or more dialogue events, each dialogue event including one or more input instances and one or more corresponding output instances or actions; An AI manager, operably coupled to the dialogue manager, applies received dialogue events to a learning program, the application including the learning program, to: Interpret the one or more input instances and output instances or actions, including representing one or more dialogue events in the model; Identify one or more knowledge gaps associated with the instances being explained; Measure the distance between subgraphs in the model that have common node labels; Utilize the structure and functionality of one or more corresponding models to compare content and identify any similarities or differences corresponding to conceptual relationships; Provides an ontology representation of any similarity or difference corresponding to conceptual relationships; and Dynamically generate one or more alternative knowledge items to bridge the knowledge gap; and A mentor optimizes the automated virtual dialogue agent based on the dialogue events, the optimization corresponding to bridging knowledge gaps to improve the performance of the dialogue system.

2. The computer system of claim 1, further comprising the learning program analyzing multiple pairs of inputs and outputs of the dialogue system, identifying one or more common features among the multiple pairs, and using the identified one or more common features to request knowledge fragments.

3. The computer system according to claim 2 further includes the AI ​​manager updating the corresponding domain knowledge with the explanation and the requested knowledge fragment.

4. The computer system according to claim 3 further includes the AI ​​manager receiving real data to guide the domain knowledge.

5. The computer system according to any one of the preceding claims, wherein, The interpretation of the one or more input instances also includes using the model to compare the text of the question-answer pairs and determine any similarities and differences in conceptual relationships.

6. The computer system according to claim 5, wherein, The dynamic request for knowledge fragments that expand the explanation also includes the AI ​​manager generating one or more questions from the determined similarities and differences, presenting the generated one or more questions through the dialogue system, and receiving at least one answer to the generated one or more questions, the answer serving as an indicator of the identified knowledge gaps.

7. The computer system according to any one of claims 1 to 4, wherein, The measured distance corresponds to the complexity of the identified knowledge gap.

8. The computer system according to any one of claims 1 to 4, wherein, The identification of the knowledge gap includes the AI ​​manager determining when the one or more input instances include one or more concepts that are not present in the one or more corresponding action output instances.

9. The computer system according to any one of claims 1 to 4 further includes the AI ​​manager requesting a selection of at least one of one or more generated alternative knowledge items, the selection bridging the identified knowledge gap.

10. A computer program product supporting active learning, the computer program product comprising a computer-readable storage medium having program code embodied therein, the program code being executable by a processor to: Natural language (NL) received in relation to the interaction with the automated virtual dialogue agent of the dialogue system, wherein the NL includes one or more dialogue events, and each dialogue event includes one or more input instances and one or more corresponding output instances or output actions; The received dialogue events are applied to an artificial intelligence (AI) platform to: Interpret the one or more input instances and output instances, including representing one or more dialogue events in the model; Identify one or more knowledge gaps associated with the instances being explained; Measure the distance between subgraphs in the model that have common node labels; Utilize the structure and functionality of one or more corresponding models to compare content and identify any similarities or differences corresponding to conceptual relationships; Provides an ontology representation of any similarity or difference corresponding to conceptual relationships; as well as Dynamically generate one or more alternative knowledge items to bridge the knowledge gap; as well as The automatic virtual dialogue agent is optimized based on the dialogue events corresponding to the bridged knowledge gaps to improve the performance of the dialogue system.

11. The computer program product of claim 10, further comprising program code for analyzing multiple pairs of inputs and outputs of the dialogue system, identifying one or more common features among the multiple pairs, and requesting knowledge fragments using the identified one or more common features.

12. The computer program product of claim 11 further includes program code that enriches the corresponding domain knowledge with the interpretation and the requested knowledge fragment.

13. The computer program product according to any one of claims 10 to 12, wherein, The program code that interprets the one or more input instances also includes instructions for using the model to compare the text of the correct question-answer and to determine any similarities and differences in conceptual relationships.

14. The computer program product according to claim 13, wherein, The program code requesting the knowledge fragment also includes generating one or more questions from the determined similarities and differences, presenting the generated one or more questions through the dialogue system, and receiving instructions to receive at least one answer to the generated one or more questions, the answer serving as an indicator of the identified knowledge gap.

15. The computer program product according to any one of claims 10 to 12, wherein the measured distance corresponds to the complexity of the identified knowledge gap.

16. A computer-implemented method for improving the performance of a dialogue system, the method comprising: The processor of the computing device receives natural language (NL) related to the interaction with the automated virtual dialogue agent of the dialogue system. The NL includes one or more dialogue events, each dialogue event including one or more input instances and one or more corresponding output instances or output actions. The processor applies the received dialogue events to an artificial intelligence (AI) platform that includes a learning program, the application of which includes: Interpret the one or more input instances and output instances or actions, including representing one or more dialogue events in the model; Measure the distance between subgraphs in the model that share common node labels; Utilize the structure and functionality of one or more corresponding models to compare content and identify any similarities or differences corresponding to conceptual relationships; Provides an ontology representation of any similarity or difference corresponding to conceptual relationships; Identify one or more knowledge gaps associated with the instance or action being explained; and Dynamically generate one or more alternative knowledge items to bridge the knowledge gap; and The automated virtual dialogue agent is optimized based on the dialogue events corresponding to the bridged knowledge gaps to improve the performance of the dialogue system.

17. The computer-implemented method of claim 16, further comprising the learning program analyzing multiple pairs of inputs and outputs of the dialogue system, identifying one or more common features among the multiple pairs, and using the identified one or more common features to request knowledge fragments.

18. The computer-implemented method of claim 17 further includes enriching the corresponding domain knowledge with the interpretation and the requested knowledge fragment.

19. The computer-implemented method of claim 16, further comprising receiving real-world data to guide the domain knowledge.

20. The computer-implemented method according to any one of claims 16 to 19, wherein, Interpreting the one or more input instances also includes using the model to compare the text of the correct question answer and to determine any similarities and differences in conceptual relationships.

21. The computer-implemented method of claim 20, wherein requesting the knowledge fragment further comprises generating one or more questions from the determined similarities and differences, presenting the generated one or more questions through the dialogue system, and receiving at least one answer to the generated one or more questions, the answer serving as an indicator of the identified knowledge gap.

22. The computer-implemented method according to any one of claims 16 to 19, wherein, The measured distance corresponds to the complexity of the identified knowledge gap.

23. A computer system, comprising: An artificial intelligence (AI) platform that communicates with a processor, the AI ​​platform having tools to guide the performance of a dialogue system, the tools including: A dialogue manager receives natural language (NL) related to interactions with the automated virtual dialogue agent of the dialogue system, the NL including one or more dialogue events, each dialogue event including one or more input instances and one or more corresponding output instances; An AI manager, operably coupled to the dialogue manager, applies received dialogue events to a learning program, the application including the learning program, to: The model represents one or more dialogue events, and is used to compare the text of the question-and-answer pair and determine one or more similarities or differences in terms of conceptual relationships. Identify the knowledge gap between the one or more input instances and the one or more corresponding output instances; Measure the distance between subgraphs in the model that share common node labels; Utilize the structure and functionality of one or more corresponding models to compare content and identify any similarities or differences corresponding to conceptual relationships; Provides an ontology representation of any similarity or difference corresponding to conceptual relationships; and Dynamically generate one or more alternative knowledge items to bridge the knowledge gap; and The instructor optimizes the automated virtual dialogue agent, the optimization corresponding to bridging the knowledge gap.

Images