Techniques for building knowledge graphs within limited knowledge domains

Abstract

To provide a method and a system for constructing a customized knowledge graph.SOLUTION: The method generates a seed graph by extracting entities and relationships between the entities from the user dataset based on a specific rule. The large knowledge graph is then traversed using a finite state machine to identify candidate entities and / or relationships to add to the seed graph. A priority function is used to select entities and / or relations from the candidate entities and / or relations. The selected entities and / or relationships are then added to the seed graph to generate the customized knowledge graph.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority to U.S. Provisional Patent Application No. 62 / 765,005, filed August 16, 2018, entitled "Techniques For Building A Knowledge Graph In Limited Knowledge Domains," and U.S. Patent Application No. 16 / 542,017, filed August 15, 2019, entitled "TECHNIQUES FOR BUILDING A KNOWLEDGE GRAPH IN LIMITED KNOWLEDGE DOMAINS," the disclosures of which, including any supplements thereto, are incorporated herein by reference in their entireties for all purposes. [Background technology]

[0002] background Knowledge graphs organize information structurally by explicitly describing the relationships between entities. Knowledge graphs typically represent a knowledge base using a directed graph with labeled edges (representing relationships) between nodes or vertices (representing entities). Knowledge graphs can be used for search, question answering, conversational interfaces (e.g., chatbots), recommender systems, and so on. Several large-scale, comprehensive knowledge graphs are available. However, natural language understanding (NLU) services used for specific applications or environments, such as building conversational interfaces (e.g., chatbots) for different users, generally perform better within a specific domain. Creating an accurate knowledge graph for a specific knowledge domain is a challenging task, due in part to the subtleties and ambiguities of natural language and limited knowledge in a specific domain, typically due to the small number of samples in user datasets. Summary of the Invention

[0003] Quick Overview The present disclosure relates generally to building customized knowledge graphs, and more specifically to techniques for constructing customized knowledge graphs for specific knowledge domains (e.g., for specific clients or specific bots) based on small user datasets and large-scale reference (e.g., external) knowledge graphs. The customized knowledge graphs can be used, for example, to improve intent classification in chatbots based on knowledge graph embedding techniques. Various embodiments of the invention are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions executable by one or more processors, and the like. [Means for solving the problem]

[0004] In certain embodiments, a computer-implemented method for generating a customized knowledge graph for an application is disclosed. The method may include receiving a user dataset for the application, extracting entities from the user dataset, identifying links between the entities based on the user dataset, and creating a seed graph representing the entities and the links between the entities. The method also includes identifying weakly connected components in the seed graph, for each weakly connected component in the seed graph, mapping entities in the weakly connected component to vertices in a reference knowledge graph, and traversing the reference knowledge graph from the mapped vertices in the reference knowledge graph based on a finite state machine. The method may include identifying up to a first threshold number of entities in the reference knowledge graph by searching the reference knowledge graph for up to a first threshold number of entities, and saving Resource Description Framework (RDF) entries of the identified entities in a buffer. The method may further include calculating priority scores for the entries in the buffer, selecting a first set of entries from the entries in the buffer having the highest priority scores, and generating a version of the customized knowledge graph by adding the entities and links identified by the first set of entries to a seed graph.

[0005] In some embodiments, the user dataset may include a plurality of user utterances. Extracting entities and identifying links between entities may include, for example, performing part-of-speech tagging, named entity recognition, and / or phrase structure analysis on the user dataset. Mapping entities in the weakly connected components to vertices in the reference knowledge graph may include, for example, using Dexter2 to map entities to vertices in the reference knowledge graph.

[0006] In some embodiments, the finite state machine may include a disambiguation state, an entity state, a forbidden state, and an end state. Traversing the reference knowledge graph may include entering the disambiguation state when the next vertex in the reference knowledge graph is a disambiguation vertex; entering the forbidden state when the next vertex in the reference knowledge graph is a forbidden vertex if the current state of the finite state machine is not the forbidden state; entering the entity state when the next vertex in the reference knowledge graph is an entity vertex and saving the RDF of the next vertex in the reference knowledge graph as an entry in a buffer; and entering the end state when the number of entries in the buffer is greater than a second threshold number. The forbidden vertex may include, for example, a date, a time value, a link to another reference knowledge graph, a web page, or a duplicated or redirected entity.

[0007] In some embodiments, the priority score of an entry in the buffer may be determined using a priority function that is a function of the number of weakly connected components in the knowledge graph, including the seed graph and the entities associated with the entry, the graph density of the knowledge graph, including the seed graph and the entities associated with the entry, and / or a control parameter. In some embodiments, the method may also include, prior to calculating the priority score of the entry in the buffer, performing reservoir sampling on the entries in the buffer to reduce the number of entries in the buffer.

[0008] In some embodiments, the method may also include updating the control parameter after adding the entities and links identified in the first set of entries to the seed graph. Updating the control parameter may include multiplying the control parameter by a value less than one. In some embodiments, the method may further include, for each weakly connected component in the customized version of the knowledge graph, mapping entities in the weakly connected component to vertices in a reference knowledge graph, identifying up to a first threshold number of entities in the reference knowledge graph by traversing the reference knowledge graph from the mapped vertices in the reference knowledge graph based on a finite state machine, and saving the RDFs of the identified entities as entries in a buffer. The method also includes calculating priority scores for the entries in the buffer using the priority function and the updated control parameter; and selecting from the entries in the buffer the entities with the highest priority. and generating an updated customized knowledge graph by adding entities and links identified by the second set of entries to the seed graph.

[0009] In some embodiments, the method may also include receiving an input utterance, extracting entities from the input utterance, generating a knowledge graph embedding of the extracted entities based on the customized knowledge graph, and classifying the input utterance based on the knowledge graph embedding of the extracted entities. In some embodiments, generating the knowledge graph embedding may include generating the knowledge graph embedding using TransE or an extension of TransE. In some embodiments, generating the knowledge graph embedding may include generating the knowledge graph embedding using TransE and a weighted scoring function. In some embodiments, generating the knowledge graph embedding may include pre-training a knowledge graph embedding of the entities extracted from the input utterance using a reference knowledge graph, and re-training the knowledge graph embedding of the entities extracted from the input utterance using the customized knowledge graph and the pre-trained knowledge graph embedding.

[0010] In some embodiments, classifying the input utterance based on the knowledge graph embeddings of the extracted entities may include determining an average knowledge graph embedding of the knowledge graph embeddings of the extracted entities, determining an average GloVe vector of the GloVe vectors of the words in the input utterance, and classifying the input utterance based on the average knowledge graph embedding and the average GloVe vector. In some embodiments, classifying the input utterance based on the average knowledge graph embedding and the average GloVe vector may include combining the average knowledge graph embedding and the average GloVe vector to generate a concatenated vector, performing principal component analysis on the concatenated vector, and classifying the input utterance based on the principal component analysis. In some embodiments, determining the average GloVe vector of the GloVe vectors of the words in the input utterance may include cleaning the input utterance, generating a GloVe vector for each word in the cleaned input utterance, and averaging the GloVe vectors for each word in the cleaned input utterance.

[0011] According to particular embodiments, a non-transitory computer-readable medium can store a plurality of instructions executable by one or more processors, which, when executed by the one or more processors, can cause the one or more processors to perform a process that may include receiving a user dataset for an application, extracting entities from the user dataset, identifying links between the entities based on the user dataset, and creating a seed graph representing the entities and the links between the entities. The process may also include identifying weakly connected components in the seed graph, and for each weakly connected component in the seed graph, mapping entities in the weakly connected component to vertices in a reference knowledge graph, identifying up to a first threshold number of entities in the reference knowledge graph by traversing the reference knowledge graph from the mapped vertices in the reference knowledge graph based on a finite state machine, and saving Resource Description Framework (RDF) entries of the identified entities in a buffer. The process may further include calculating priority scores for the entries in the buffer, selecting a first set of entries from the entries in the buffer having the highest priority scores, and generating a version of the customized knowledge graph by adding entities and links identified in the first set of entries to a seed graph.

[0012] According to certain embodiments, a system may include one or more processors and a memory coupled to the one or more processors. The memory may store instructions that, when executed by the one or more processors, cause the system to perform operations including receiving a user dataset for an application, extracting entities from the user dataset, identifying links between the entities based on the user dataset, and creating a seed graph representing the entities and the links between the entities. The operations may also include identifying weakly connected components in the seed graph, and for each weakly connected component in the seed graph, mapping entities in the weakly connected component to vertices in a reference knowledge graph, identifying up to a first threshold number of entities in the reference knowledge graph by traversing the reference knowledge graph from the mapped vertices in the reference knowledge graph based on a finite state machine, and saving Resource Description Framework (RDF) entries of the identified entities in a buffer. The operations may further include calculating priority scores for the entries in the buffer, selecting a first set of entries from the entries in the buffer having the highest priority scores, and generating a version of the customized knowledge graph by adding entities and links identified in the first set of entries to a seed graph.

[0013] The foregoing, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0014] Illustrative examples will now be described in detail with reference to the following drawings: [Brief explanation of the drawings]

[0015] [Figure 1]FIG. 1 illustrates a distributed system implementing a bot system for interacting with end users using messaging applications, according to certain embodiments. [Figure 2] FIG. 1 illustrates an integrated system including a bot system and a bot analytics system for monitoring, analyzing, visualizing, and improving the performance of the bot system, according to certain embodiments. [Figure 3] FIG. 1 is a diagram illustrating an example of a knowledge graph. [Figure 4A] FIG. 1 is a diagram illustrating an example of a process for constructing a knowledge graph. [Figure 4B] FIG. 1 is a diagram illustrating an example of a process for constructing a knowledge graph. [Figure 4C] FIG. 1 is a diagram illustrating an example of a process for constructing a knowledge graph. [Figure 4D] FIG. 1 is a diagram illustrating an example of a process for constructing a knowledge graph. [Figure 5] FIG. 1 is a simplified flowchart illustrating an example method for building a customized knowledge graph for a particular domain (e.g., for a chatbot), according to certain embodiments. [Figure 6] FIG. 2 illustrates an example of a finite state machine (FSM) for controlling the traversal of a reference knowledge graph, according to certain embodiments. [Figure 7] FIG. 1 illustrates an example of a seed graph generated based on a user dataset (e.g., user utterances), according to certain embodiments. [Figure 8] FIG. 8 illustrates an example of a customized knowledge graph generated based on the seed graph shown in FIG. 7, according to certain embodiments. [Figure 9] FIG. 9 is an expanded view of a portion of the customized knowledge graph shown in FIG. 8. [Figure 10] 1 shows a simplified diagram of a distributed system for implementing certain embodiments. [Figure 11]FIG. 1 is a simplified block diagram of a cloud-based system environment in which various services may be provided as cloud services, according to certain embodiments. [Figure 12] FIG. 1 illustrates an example of a computer system that may be used to implement certain embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0016] Detailed Description The present disclosure relates generally to building customized knowledge graphs, and more specifically to techniques for constructing a customized knowledge graph for a specific knowledge domain based on limited knowledge within the domain (e.g., a small user dataset) and a large-scale reference (e.g., external) knowledge graph. The customized knowledge graph can be used, for example, to improve intent classification in chatbots based on knowledge graph embedding techniques. Various embodiments of the invention are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions executable by one or more processors, and the like.

[0017] Natural language understanding (NLU) services used in specific applications or environments, such as conversational interfaces for different users (e.g., chatbots), may work more effectively with a customized knowledge graph within a specific domain rather than a large, all-encompassing knowledge graph. For example, a customized knowledge graph may be less noisy, and therefore it may be more efficient and accurate to use a small, customized knowledge graph rather than a large, general-purpose knowledge graph. However, the knowledge available within a specific domain may be limited, and therefore a customized knowledge graph generated based on limited knowledge may be sparse.

[0018] According to an embodiment, entities can be extracted from a small user dataset (e.g., including user utterances, intents, entities, and questions and answers) for a custom application (e.g., a chatbot) based on specific rules. A seed graph can be generated based on the extracted entities and the connections or relationships between the entities. The seed graph can then be extended into a customized knowledge graph for the custom application by traversing a large reference knowledge graph, such as WikiData®, using a finite state machine to identify candidate entities and / or relationships to add to the seed graph. This traversal can also help identify possible additional relationships and related facts between entities in the user dataset from the reference knowledge graph. A scoring function can be used to select entities and / or relationships from the identified entities and relationships, and the selected entities and / or relationships can then be added to the seed graph to generate a customized knowledge graph. The generated customized knowledge graph can be used for a customer application, such as a custom chatbot, using specific knowledge graph embedding techniques. Ablation studies show that additional semantic information captured by a customized knowledge graph can improve the performance of custom chatbots. The techniques disclosed herein can also be used for other applications, such as question answering or relation extraction.

[0019] As used herein, "chatbot," "bot," or "skill" refers to a computer program designed to simulate, among other things, a conversation with a human user over the Internet. Individual skills can be designed to interact with a user and perform specific types of tasks, such as ordering food, making reservations, and changing contact information. Each skill can assist a user in completing a task through a combination of text messages and simple UI elements such as selection lists.

[0020] As used herein, the term "intent" may refer to a category of action or task that a user expects a skill to perform on their behalf. The term "entity" may refer to a variable that identifies key information from a user that enables a skill to accomplish a task. The term "component" may refer to various functions that a skill can use to respond to a user. The term "component" may refer to a general function, such as outputting text or returning information from a backend and executing custom logic. The term "dialog flow" may refer to a definition of a skill's interaction with a user and may describe how a skill responds and behaves according to user input. The term "channel" may refer to a platform-specific configuration that allows a skill to access a messaging platform or client messaging application. A skill can have several channels configured for it, allowing it to run simultaneously on different services or platforms.

[0021] As used herein, an utterance or message may refer to one or more sentences exchanged during a conversation, and a conversation may refer to a communication session that may include one or more utterances or messages. A conversation may include one or more stages or states. A conversation flow may be an abstraction of multiple conversations that include the same stages or states and the same transitions from stage to stage. Each conversation may be a specific instance of a corresponding conversation flow. A state or stage of a conversation or conversation flow may be associated with a state of a state machine that the bot system maintains to conduct conversations with other bot systems or people. In some cases, a state may correspond to an end user's intent or purpose. An end user, as used herein, may refer to an end user of a bot system, such as a person or another entity that participates in a conversation with the bot system through a messaging application or platform. For example, an end user may be a customer or client of a company that owns the bot system. As used herein, a user of the analysis system may refer to the owner, operator, administrator, or developer of the bot system.

[0022] In the following description, for purposes of explanation, specific details are set forth to provide a thorough understanding of examples of the present disclosure. However, it will be apparent that various examples may be practiced without these specific details. The following description provides examples only and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of these examples will provide those skilled in the art with an enabling description for implementing the examples. It should be understood that various changes in the function and arrangement of elements can be made without departing from the spirit and scope of the present disclosure as set forth in the appended claims. The drawings and description are not intended to be limiting. Circuits, systems, networks, processes, and other components may be shown in block diagram form as components so as not to obscure the examples with unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as not to obscure the examples. The teachings disclosed herein may also be applied to various types of applications, such as mobile applications, non-mobile applications, desktop applications, web applications, enterprise applications, etc. Furthermore, the teachings of the present disclosure are not limited to any particular operating environment (eg, operating system, device, platform, etc.), but instead can be applied to a number of different operating environments.

[0023] The examples may also be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a process may be described as sequential, many of the operations may be performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

[0024] The words "exemplary" or "exemplary" are used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" or "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0025] The terms "machine-readable storage medium" or "computer-readable storage medium" include, but are not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instruction(s) and / or data. Machine-readable storage medium or computer-readable storage medium may include non-transitory media capable of storing data, but does not include carrier waves and / or transitory electronic signals propagated wirelessly or via wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, memory, or memory devices. A computer program product may include code and / or machine-executable instructions, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment can be coupled to another code segment or a hardware circuit by sending and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0026] Furthermore, the examples may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented by software, firmware, middleware, or microcode, the program code or code segments (e.g., a computer program product) to perform the necessary tasks may be stored on a machine-readable medium. A processor may perform the necessary tasks.

[0027] The systems shown in some of the figures may be provided in a variety of configurations, hi some examples, the systems may be configured as distributed systems in which one or more components of the systems are distributed across one or more networks in a cloud computing system.

[0028] Where a component is described as being "configured to" perform particular operations, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform those operations, or any combination thereof, by programming or controlling the electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry).

[0029] Many users around the world are on instant messaging or chat platforms to get quick reactions. Organizations often use these instant messaging platforms to Using messaging or chat platforms, organizations have live conversations with customers (or end users). However, hiring service personnel to interact with customers or end users live can be prohibitively expensive for organizations. Therefore, chatbots (also known as bots, skills, chatterbots, or talkbots) have begun to be developed specifically to simulate conversations with end users over the internet. End users can interact with bots through messaging applications that the end users already have installed and using. Intelligent bots, typically driven by artificial intelligence (AI), can communicate more intelligently and contextually in live conversations, enhancing the conversational experience by enabling a more natural conversation between the bot and the end user. Instead of end users having to memorize a fixed set of keywords or commands that the bot knows how to respond to, intelligent bots can understand the end user's intent based on the user's natural language utterances and respond accordingly.

[0030] A bot system may include one or more user intent classification engines for identifying an end user's intent based on the end user's utterances, and one or more dialog engines for intelligently and contextually responding to the user's utterances according to the identified end user's intent. However, building a bot system, such as a user intent classification engine, that can determine an end user's intent based on the user's utterances is a challenging task, due in part to the subtleties and ambiguities of natural language, the dimensions of the input space (e.g., possible user utterances), and the size of the output space (number of intents).

[0031] Bots are typically capable of responding to natural language messages (e.g., questions or comments) through messaging applications that use natural language messages. Enterprises may use one or more bot systems to interact with end users through messaging applications. The messaging applications, which may be referred to as channels, may be messaging applications that end users already have installed and are familiar with, and that are preferred by end users. Thus, end users do not need to download and install new applications to chat with a bot system. Messaging applications may include, for example, over-the-top (OTT) messaging channels (such as Facebook® Messenger, Facebook WhatsApp, WeChat®, Line, Kik, Talk, Skype®, Slack, or SMS), virtual private assistants (such as Amazon® Dot, Echo, or Show, Google Home®, Apple® HomePod), mobile and web application extensions that extend native or hybrid / responsive mobile or web applications with chat capabilities, or voice-based input (such as devices or applications with interfaces that use Siri®, Cortana®, Google Voice, or other voice input for interaction).

[0032] In some examples, a bot system may be associated with a Uniform Resource Identifier (URI). The URI may identify the bot system using a string of characters. The URI may act as a webhook for one or more messaging application systems. A URI may be used. A URI may include, for example, a Uniform Resource Locator (URL) or a Uniform Resource Name (URN). The bot system may be designed to receive a message (e.g., a HyperText Transfer Protocol (HTTP) PostCall message) from a messaging application system. The HTTP PostCall message may be directed to a URI from the messaging application system. In some embodiments, the message may be an HTTP PostCall message. For example, a bot system may receive a message via Short Message Service (SMS). While the description herein refers to communications received by a bot system as a message, those skilled in the art will understand that the message may be an HTTP post call message, an SMS message, or any other type of message between two systems.

[0033] End users can interact with bot systems through conversational interactions (sometimes called conversational user interfaces (UIs)), similar to interactions between people. In some cases, this interaction begins with the end user saying "Hello" to the bot and the bot responding. The interaction may include the bot responding with "Hi" and asking the end user how they would like to proceed. In some cases, this interaction may be a transactional interaction with a banking bot, for example, transferring money from one account to another, an informational interaction with an HR bot, for example, checking vacation balances, or an interaction with a retail bot, for example, discussing returning a purchased item or seeking technical support.

[0034] In some embodiments, the bot system can intelligently handle interactions with end users without interaction with an administrator or developer of the bot system. For example, an end user can send one or more messages to the bot system to achieve a desired goal. The messages can include specific content, such as text, emojis, audio, images, video, or other message delivery methods. In some embodiments, the bot system can then send this content in a standard form (e.g., a representational state transfer to an enterprise service with appropriate parameters). ) (REST) ​​calls) and generate natural language responses. The bot system can also prompt the end user for additional input parameters or request other additional information. In some embodiments, the bot system can also initiate a dialogue with the end user rather than passively responding to the end user's utterances.

[0035] A conversation with a bot can follow a specific conversational flow that includes multiple states. This flow can define what happens next based on input. In some embodiments, a bot system can be implemented using a state machine that includes user-defined states (e.g., end-user intent) and actions to perform at or for each state. The conversation can take different paths based on end-user input, which can affect the decisions the bot makes about this flow. For example, at each state, the bot can determine the end-user's intent to determine the appropriate action to take next based on the end-user's input.

[0036] Intents may include goals that an end user wants to achieve. Intents map end user input to actions that a backend system should perform on the end user's behalf. Thus, a bot maps end user utterances to specific use cases or jobs, such as ordering a pizza, getting an account balance, transferring money, making a purchase, or returning a product, based on phrases spoken by the end user in natural language. Human conversations are often nonlinear in nature. End users often branch into different states during a conversation. For example, if an end user wants to transfer funds from account A to a recipient, they may begin a conversation with the bot system by asking the bot to, for example, pay for dinner for the recipient. The bot may respond with, for example, "From which account?" The end user may choose a checking account but be unsure of the balance of this account. Therefore, the end user may switch context and ask for the balance, latest transaction, etc. In other words, the end user may trigger a flow and state change, for example, from transferring funds to checking the balance, and then to the latest transaction. At some point, the end user may decide to return to their original intention of making a payment to the recipient. Thus, one task of the bot system is to The goal is to dynamically determine the end user's intentions from natural language utterances.

[0037] The bot is powered by a natural language processing (NLP) engine and and / or machine learning models (e.g., intent classifiers) can be used to map end-user utterances to specific intents. For example, a machine learning-based NLP engine can learn to understand and classify end-user natural language speech and extract necessary information from the speech to perform precise actions, such as executing a transaction or examining data from recorded back-end systems.

[0038] 1 illustrates a distributed system 100 that may be used to implement a bot system for interacting with end users using messaging applications, according to certain embodiments. System 100 may include a bot system 120, one or more messaging application systems 115, and one or more end user devices, such as one or more mobile devices 110. In some examples, the messaging application may be installed on an electronic device (e.g., a desktop computer, a laptop, a mobile device 110, etc.). While the description herein refers to a mobile device and messaging application, those skilled in the art will understand that any electronic device may be used and any messaging platform or messaging application may be used, such as FACEBOOK® Messenger, WHATSAPP® instant messaging software, WECHAT® mobile text and voice messaging communication service, KIK® Messenger, TELEGRAM® Messenger, SKYPE MOBILE® Messenger, short message service (SMS), or any other messaging application that provides a platform for end users to communicate. In other examples, messaging applications may be executed through a browser (e.g., GOOGLE CHROME® browser, MOZILLA® FIREFOX® browser, and INTERNET EXPLORER® browser) installed on mobile device 110. In some embodiments, two or more messaging applications may be installed on an end user device to communicate through two or more messaging platforms (e.g., two or more messaging application systems 115).

[0039] The messaging application may be facilitated by a messaging platform, such as messaging application system 115. The mobile device 110 may be connected to the messaging application system 115 by a first network (e.g., the Internet). The messaging application system 115 may be a messaging platform provided by a third party, such as Facebook, Tencent, Google®, Microsoft®, etc. The messaging application system 115 may manage content sent and received through messaging applications across multiple mobile devices or other end-user devices.

[0040] The bot system 120 (e.g., implemented on one or more servers) may also be communicatively coupled to the messaging application system 115 to send and receive messages. Communication between the messaging application system 115 and the bot system 120 may occur over a second network (e.g., the Internet). The first network and the second network may be the same network, or may be similar or entirely different networks. The messaging application system 115 can route content (e.g., messages or information from messages) from the mobile device 110 to the bot system 120 using the Internet. In some embodiments, the destination of the content (e.g., the ID of the bot system 120) may be included in the content as a nominal address. In some embodiments, the bot system 120 may be configured to communicate with more than one messaging application system 115.

[0041] As previously mentioned, content exchanged between end users or between end users and the bot system may include, for example, text, emojis, audio, media (e.g., photos, videos, links), or any other method of messaging. An example of a message that the bot system 120 receives from, for example, Facebook Messenger, may include:

[0042]

number

[0043] Bot system 120 can receive content from messaging application system 115 using connector 130, which acts as an interface between messaging application system 115 and bot system 120. In some embodiments, connector 130 may standardize content from messaging application system 115 so that bot system 120 can parse content across various messaging application systems. The content standardization process may include formatting content from each type of messaging application into a common format for processing. In some embodiments, bot system 120 can receive content from messaging applications (such as FACEBOOK® Messenger, WHATSAPP® instant messaging software, WECHAT® mobile text and voice messaging communication service, KIK® Messenger, TELEGRAM® Messenger, SKYPE MOBILE® Messenger, and others). A connector 130 may include one or more connectors for each of the following types of content: message queues (e.g., jar, short message service (SMS), etc.). In some implementations, a connector 130 may route content to a message in-queue 140. The message in-queue 140 may include a buffer (e.g., a first-in, first-out (FIFO) buffer) that stores content in the order in which it is received. In some embodiments, each connector 130 may be associated with one or more message in-queues.

[0044] The message in-queue 140 sends content to the message processor 150 as it becomes available. In some embodiments, the message processor 150 can pull content from the message in-queue 140. As described in more detail below, the message processor 150 can parse messages and determine the intent of the parsed messages. In some embodiments, the message processor 150 can include a natural language processor 152 and an intent determination subsystem 154. The natural language processor 152 can parse messages and perform certain semantic analysis, such as identifying subjects, predicates (e.g., actions), and / or objects. The intent determination subsystem 154 can determine the end user's intent based on the parsed messages. As previously mentioned, the intent can include the purpose of the message. For example, the purpose of the message can include ordering a pizza, ordering a computer, transferring money, inquiring about delivery, etc. In some embodiments, parameters associated with the intent, sometimes called entities, that more specifically define or clarify the action to be taken, may also be extracted from the message by the natural language processor 152 and / or intent determination subsystem 154.

[0045] After message processor 150 determines the end user's intent based on the content, it can send the determined intent (and parameters associated with the intent) to action engine 160. Action engine 160 can be used to determine an action to perform based on the intent (and parameters associated with the intent) and the current state (or context) of the state machine described above. For example, action engine 160 can send specific outbound content as a response to message outqueue 170 and / or send commands to or retrieve information from an enterprise service, such as enterprise service 125. Message outqueue 170 can send the outbound content to connector 130. Connector 130 can then send the outbound content to a messaging application system indicated by action engine 160, which may be the same as or different from messaging application system 115. Messaging application system 115 can then forward the outbound content to a messaging application on mobile device 110.

[0046] The bot system 120 may communicate with one or more enterprise services (e.g., enterprise service 125), or may communicate with one or more storage systems to store and / or parse messages received from the bot system 120, or may communicate with a content system to provide content to the bot system 120. The enterprise service 125 may communicate with one or more of the connectors 130, the action engine 160, or any combination thereof. The enterprise service 125 may communicate with the connectors 130 similar to communication with the messaging application system 115. The enterprise service 125 may communicate with the connectors 130 to be associated with one or more end users. The enterprise service 125 can send content to the connector 130, which can then cause the bot system 120 to perform an action associated with the end user. The action engine 160 can communicate with the enterprise service 125 to obtain information from the enterprise service 125 and / or instruct the enterprise service 125 to perform an action identified by the action engine 160.

[0047] In some embodiments, bot system 120 may include one or more timers. The timers may cause action engine 160 to send content using connectors 130 and messaging application system 115 after a certain amount of time has elapsed. In some embodiments, the timers may send content to bot system 120, similar to how an end user or enterprise service 125 would. For example, the timers may send messages to bot system 120 to be analyzed, such as messages from end users.

[0048] In one particular embodiment, an end user can use a mobile device 110 to send a message to the bot system 120 through the messaging application system 115. The message may include a greeting such as "Hello" or "Hi." The bot system may determine that a new conversation with the end user has begun and start a state machine. In some embodiments, the bot system may identify one or more characteristics of the end user. For example, the bot system may identify the end user's name using a profile associated with the end user on a messaging application system. The bot system may respond to the end user on the messaging application using the one or more characteristics. The response may include a message addressed to the end user in response to a message received from the end user. For example, the response may be, "Hi Tom, What can I do for you?" The message may include a greeting using the end user's name, such as "Tom, how can I help you?" Depending on the enterprise associated with the bot system, the bot system may evolve to meet the goals of that enterprise. For example, if the bot system is associated with a pizza delivery service, the bot system may send a message to the end user asking if the end user would like to order a pizza. A conversation between the bot system and the end user will then begin and continue until the bot system ends the conversation or the end user stops responding to the bot system.

[0049] In some embodiments, a bot system may initiate a conversation with an end user. A conversation initiated by the bot system may be a response to a previous conversation with the end user. For example, an end user may have ordered a pizza in a previous conversation. The bot system may then initiate a conversation when the pizza is ready. In some embodiments, the bot system may determine that the pizza is ready when it receives an indication from a business associated with the bot system (e.g., an employee sends a message to the bot system that the pizza is ready). The conversation may include a message sent to the end user indicating that the pizza is ready.

[0050] In some embodiments, the bot system may send a message to an end user in a messaging application that is different from the messaging application in which the previous message was received. For example, the bot system may decide to send a message using Short Message Service (SMS) rather than Facebook Messenger. In such implementations, the bot system may integrate multiple messaging applications.

[0051] In some embodiments, the bot system may determine to start a conversation based on a timer. For example, the bot system may determine that a user has a one-week timer since a pizza was ordered. When the one-week timer expires, the bot system may start a conversation with the end user to order another pizza. This timer may be configured by the enterprise and implemented by the bot system.

[0052] As previously mentioned, in some embodiments, the action engine 160 may send commands to or retrieve information from some enterprise services 125. For example, if the bot system 120 (more specifically, the message processor 150) determines that the intent is to check a balance, the bot system 120 may determine which of several accounts (e.g., a checking account or a savings account) the balance of which is to be checked. If the end user's input is "What's my balance in my savings account?" )”, the bot system 120 extracts “savings” and sends it to the bank server. The bot system 120 may send a command to the end user to check the balance, and then send the received balance information through a message to the end user. If the end user initially only says "what's the balance in my account," the bot system 120 may further send a message to the end user prompting the end user to specify a particular account, or it may retrieve information for all of the end user's accounts and send this account information to the end user for the end user to select.

[0053] In some embodiments, the bot system may retain information between conversations. This information may be used later to prevent the bot system from having to ask questions each time a new conversation is initiated between an end user and the bot system. For example, the bot system may store information about a previous pizza order placed by an end user. In the new conversation, the bot system may send a message to the end user asking whether the end user would like the same order as the previous one.

[0054] In some embodiments, the bot system 120 may store information associated with an end user in a cache. The cache may write to a database to store the information after an outbound message is sent from the connector 130 to the messaging application system. In other embodiments, the cache may write to the data at different times (e.g., after a specific event, after each event, after a set amount of time, or any other metric for determining when to write to the database).

[0055] The bot system 120 may scale each component when a slowdown is identified. For example, when the bot system 120 identifies that the number of messages arriving at the connector 130 exceeds a threshold, the bot system 120 may add one or more additional connectors to the connector 130. Additionally, the number of message in queues, message processors, action engine instances, and message out queues may be increased depending on where the slowdown is occurring. In such implementations, additional components may be added without adding other additional components. For example, connectors may be added without adding additional action engine instances. In some implementations, one or more components or portions of a component of the bot system 120 may run on a virtual machine. Running on a virtual machine allows additional virtual machines to be started when desired.

[0056] As previously mentioned, building a bot system, such as a user intent classification engine, that can determine an end user's intent based on the end user's utterances is a difficult task, due in part to the subtleties and ambiguities of natural language, the dimensions of the input space (e.g., possible end user utterances), and the size of the output space (number of intents). As such, new bot systems may need to be monitored, debugged, and modified to improve their performance and the user's experience with them. In many cases, it may be difficult to determine how to improve a bot system without more specifically identifying the underlying reasons why the bot system is performing less than desired and using analysis or optimization tools.

[0057] In some situations, a bot owner, developer, or administrator may want to monitor the operation of a bot to understand how the bot is being used and where end users abandon the bot in order to improve the bot. For example, a bot owner, developer, or administrator may want to know which bot conversations were successful and which were unsuccessful in order to identify and diagnose underperforming aspects of the bot system.

[0058] According to some embodiments, an analysis system may be integrated with a bot system. The analysis system may monitor events that occur during conversations between end users and the bot system, aggregate and analyze the collected events, and illustrate information about the conversations on a graphic user interface at various levels of generalization, such as all conversations, various categories of conversations, and individual conversations. For example, the graphic user interface may display options for filtering or selecting specific types of conversations or individual conversations, and illustrate the selected information, for example, by visualizing the conversation path. The analysis system may also provide suggestions, options, or other information for improving the bot system.

[0059] FIG. 2 illustrates an integrated system 200 including a bot system (such as bot system 120) and a bot analysis system for monitoring, analyzing, visualizing, and improving the performance of the bot system, according to certain embodiments. As shown, the bot system may include a connector 230 and multiple bot engines, such as a dialog engine 212, an intent modeler 214, an entity resolver 216, and custom components 218. The bot system may also include a database 240, an administration API 250, a user interface 254, and a UI server 252. The bot analysis system may include a collector 255, an enrichment engine 260, a database 270, and a REST server 280. The bot analysis system may also include a user interface 292 and a UI server 290. The collector 255 of the bot analysis system may collect events 205 that occur in the bot system. Feedback 294 from the bot analysis system may be provided to the bot system through the user interface 292 and the user interface 254.

[0060] Connector 230 can act as an interface between the bot system and one or more end users through one or more channels, such as channels 220 and 222. Each channel can be a messaging channel (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), a virtual private assistant (such as Amazon Dot, Echo or Show, Google Home, Apple HomePod, etc.), a mobile and web application extension that extends a native or hybrid / responsive mobile or web application with chat capabilities, or a voice-based input (such as Siri, Cobra, etc.). The content may be a messaging application such as a messaging app (such as a device or application with an interface that uses voice input for interaction, such as RTana, Google Voice, or other voice input for interaction). In some embodiments, connectors 130 may standardize content from various channels so that the bot system can parse the content across various messaging application systems. The content standardization process may include formatting content from each type of messaging application into a common format for processing. In some embodiments, the bot system may include one or more connectors for each channel.

[0061] The intent modeler 214 can be used to determine the end user's intent corresponding to the end user's utterance. In some embodiments, the intent modeler 214, which allows the bot system to determine the end user's intent based on one or more messages received from the end user, can use a natural language processor to tag parts of speech (verb, noun, adjective), discover lemmas / stem (runs / running / ran → run), and tag entities (Texas → location). In some embodiments, the intent modeler 214 can standardize messages. For example, "Mary ran to Texas" can be converted to "PERSON run to LOCATION" The intent modeler 214 may also include logic for detecting words with the same meaning in an end user's message. For example, if a training dataset contains "Mary ran to Texas" and "Bob walked to Detroit," both of which are likely to be mapped to the same intent and "run / walk" are considered to be in the same set of intents, the intent modeler 214 may For clarity of illustration, we may learn that run = walk. In one illustrative example, "Mary ran to Texas" may become "PERSON run to LOCATION" and "Bob walked to Detroit" may become "PERSON walk to LOCATION." In the example shown: , both of these sentences map to the same intent because, for the purposes of intent resolution, "noun run to noun" is identical to "noun walk to noun." In another example, "I want to order a large cheese pizza" and "I want to order a small pepperoni pizza" Both "I want to order a pepperoni pizza" can be standardized to "I want to order a Bots_PizzaSize Bots_Toppings pizza".

[0062] After normalization, the probability that a word occurrence represents a particular intent may be determined. In some instances, the probabilities can be combined as if they were independent using basic probability operations. For example, if "order" means there is a 20% chance of ordering a pizza. and "pizza" suggests that there is a 10% chance of ordering pizza. , the overall probability would be 1-(1-0.2)(1-0.1)=28%. Some probabilities may be based on the presence of words, or on specific linguistic elements, such as the presence of negation or personal pronouns.

[0063] Another level of rules may be template rules, which are combinations of words. In some examples, all sentences in the training dataset, once standardized, can automatically become rules. In such examples, the training dataset may contain very few short sentences. A template rule returns a probability of 1. New rules may be generated through an induction process. For example, the following sentences "How much did I spend last month on gas?" and "How much did I spend on May on food?" may belong to track spending. Using these sentences, one can derive the rule "How much did I spend" as the common part of these sentences. In other examples, the training dataset The set could achieve the same result by including the phrase "How much did I spend."

[0064] The above example allows the definition of an intent to avoid duplicates, such as variations of a specified entity (e.g., "Send money to Sue" and "Send money to Bob"). However, similar sentences that differ by one or two words can be used for training. Similar sentences allow the model to learn which words may have the same meaning and which words may be common misspellings for the purpose of intent resolution.

[0065] If a particular word or set of words (such as a verb) is important for one intent, the probability can be manipulated by having more examples of using such words (and their synonyms) and fewer examples of such words being used in other intents.

[0066] Examples can also be provided to prevent the model from making false assertions. For example, certain subphrases or words that only appear in certain intents can cause false assertions. Similarly, the model can be prevented from synthesizing broad rules using similar sentences that belong to different intents for training.

[0067] The entity resolver 216 can identify entities (e.g., objects) that correspond to the end user's intent. For example, in addition to the end user's intent identified by the intent modeler 214, such as "order pizza," the entity resolver 216 can also identify entities (e.g., objects) that correspond to the end user's intent. The resolver 216 can resolve entities associated with this intent, such as the type of pizza, toppings, etc.

[0068] The dialog engine 212 can be used to process conversations between an end user and a bot system. For example, the dialog engine 212 can respond to an utterance from an end user based on the end user's intent identified by the intent modeler 214 and the entities associated with the end user's intent identified by the entity resolver 216. In some embodiments, the dialog engine 212 can process conversations with an end user by using a state machine that includes user-defined states (e.g., end user intents) and actions to be performed in or for each state.

[0069] Custom components 218 may include modules customized for a particular bot system. For example, a financial bot may include custom components that may be used, for example, to check balances, transfer funds, or pay bills.

[0070] Database 240 can be used to store data about the bot system, such as data about classification models, conversation logs, etc. Management API 250 can be used by an administrator or developer of the bot system to manage the bot system, such as retraining classification models, editing intents, or otherwise modifying the bot system. The administrator or developer can manage the bot system using user interface 254 and UI server 252.

[0071] Various events may be generated during the execution of a bot system. These events may be generated based on one or more instructions contained in the bot system. For example, an event may be generated when the bot system enters a particular state defined by the administrator or developer of the bot system. Once events are generated, a bot analysis system may collect, store, and analyze these events. When capturing an event, additional information associated with the event may also be collected, which may indicate the current context in which the event is generated.

[0072] For example, a conversation event may be generated by the dialog engine 212. The conversation event may include a message (called msg_received) that the bot system receives from an end-user device. The msg_received may include one or more of the following parameters or variables: message content, time the bot system receives the message, language of the received message, device properties (e.g., version or name), operating system properties (e.g., version or name), geological properties (e.g., Internet Protocol address, latitude, longitude, etc.), identification information (e.g., user ID, session ID, bot system ID, tenant ID, etc.), timestamp (device-created, device-sent, collector-derived timestamp), channel, etc.

[0073] Conversation events may also include messages (called msg_sent) that the bot system sends to the end user device. The msg_sent may include one or more of the following: message content (e.g., the text or HTML of the message), the time the bot system sends the message, the language of the message, the creator of the message (e.g., the bot system or the end user device), device properties, operating system properties, browser properties (e.g., version or name), application properties (e.g., version or name), geographic properties (e.g., Internet Protocol address, latitude, longitude, etc.), identification information (e.g., user ID, session ID, bot system ID, tenant ID, etc.), channel (e.g., Facebook or Webhook), etc.

[0074] The dialog engine 212 may also generate dialog state execution events. As previously mentioned, the dialog engine 212 can determine the flow of a conversation with an end user by using a state machine. The state machine may include a set of states and rules for transitioning between states. The dialog engine 212 executes a state machine for each end user conversation, and a dialog state execution event may be generated for each state the dialog engine 212 goes through to process an end user utterance. Attributes of the dialog state execution event may include, for example, a state name, a component name, a next action, an entity match, an intent match, variables, a user query statement, a response statement, an execution time, a communication language, a device property, an operating system property, a geological property, an identification information, a timestamp, a channel, etc. The state name may be the name of the currently executed state or an "error state." The component name may be the name of the bot component to be executed for the current state. The next action may be the next action to be performed. The entity match may be the entity resolved in the current message. The intent match may be the intent resolved with a score value. The variable may be a variable value for the current state. The query statement may be a message sent by the end user. The response statement may be a message sent to the end user. The execution time may be a timestamp of the completed state execution. The communication language may be the language of the messages being exchanged. The device and / or operating system properties may correspond to the end user interacting with the bot system. The browser and / or application properties may be associated with the end user interacting with the bot system. The geological property may be the location of the end user interacting with the bot system.

[0075] An intent resolution event can occur as a result of the execution of the intent modeler 214. The system 214 may use a trained or otherwise defined classification model to identify an end-user's intent from a set of intents based on the end-user's utterance. The results of the intent classification may be captured as intent resolution event attributes, which may include, for example, a final intent classification result (e.g., an identified intent) and a confidence score associated with each intent in the set of intents.

[0076] The entity resolver 216 can generate entity resolver events. An entity is something that is associated with an end user intent. Entity definition rules may be determined when the bot system is created. For example, in addition to resolving an end user intent, such as "order a pizza," the bot system can also resolve associated entities, such as the type of pizza, toppings, etc., using the entity resolver 216. Entity resolver events may be captured in entity resolution. Examples of attributes associated with an entity resolver event may include the entity name, the rule applied, the search term, the resolved state, the query statement, the entity type, execution time, communication language, device properties, operating system properties, browser properties, application properties, geological properties, identification information, timestamp, channel, etc. The entity name may be the name of the entity currently being resolved. The rule applied may be, for example, predecessor, successor, or aggregate. The search term may be a destination, origin, etc. The property may be a search term from / to. The resolved state may be a dialog state resolved for an entity. The query statement may be a message containing a value for the entity. The entity type may be system or derived. The execution time may be a timestamp of the entity resolution. The communication language may be the language of the messages being exchanged. The device and / or operating system property may be associated with an end user interacting with the bot system. The browser and / or application property may be associated with an end user interacting with the bot system. The geological property may be the location of an end user interacting with the bot system.

[0077] Custom components may also generate events, such as predefined events or custom events. A predefined event may be a property captured in the execution of a custom component. Examples of attributes of a predefined event may include a component name, an event name, a payload, an execution time, a communication language, a device property, an operating system property, a browser property, an application property, a geological property, an identification, a timestamp, a channel, etc. The component name may be the name of the currently executing custom component. The event name may be invoked, invocation_failed, replied, replied_failed, etc. The payload may be, in the case of a failure, a reason for the failure, a stack trace, etc. The execution time may be a timestamp indicating when the event occurred. The communication language may be the language of the messages being exchanged. The device and / or operating system properties may be associated with an end user interacting with the bot system. The browser and / or application properties may be associated with an end user interacting with the bot system. The geological property may be the location of an end user interacting with the bot system.

[0078] A custom component can also emit custom events during the execution of the custom component. Examples of attributes of a custom event include the component name, the event name, the custom payload, the execution time, the communication language, the device properties, the operating system properties, the browser properties, the application properties, The information may include geological properties, identification information, timestamps, channels, etc. The component name may be the name of the custom component currently being executed. The event name may be a user-defined event name (e.g., Balance_Retrieved). The payload may be, for example, {"amount":"USD 100", "account":"checking"}. The execution time may be a timestamp indicating when the event occurred. The communication language may be the language of the messages being exchanged. The device and / or operating system properties may be associated with an end user interacting with the bot system. The browser and / or application properties may be associated with an end user interacting with the bot system. The geological property may be the location of an end user interacting with the bot system.

[0079] Error and timeout events may also be generated by the bot system during execution. An error event may be generated when an error occurs. A timeout event may be generated when an end user is not speaking for a certain period of time, which may be configured in the channel.

[0080] The bot analysis system can collect events and additional information as the bot system engages in a conversation with an end user and generates corresponding events. For example, collector 255 can collect events and additional information and send the collected information to a queue. In some embodiments, collector 255 is configurable and can be programmed to collect various events and / or event attributes as desired. For example, collector 255 can be configured to capture dialog state attributes, intent resolution attributes, entity resolution attributes, and error and timeout attributes. In some embodiments, collector 255 can also be configured to collect information about events 295 generated by systems other than the bot system.

[0081] The enrichment engine 260 can perform validation and enrichment on the collected events and other information and write them to the database 270. For example, the enrichment engine 260 can determine the location of an end user associated with a collected IP address. As another example, the enrichment engine 260 can extract certain features from the collected information, such as determining the web browser or channel used by the end user. The REST server 280 can analyze the enriched events and other information and generate various reports based on certain aggregate metrics 272. The reports can be displayed on a user interface 292 via a UI server 290 to a bot system owner, administrator, or developer. The bot system owner, administrator, or developer can provide feedback 294 to the bot system to improve it.

[0082] As mentioned previously, building a bot system, such as a user intent classification engine (e.g., message processor 150 or intent modeler 214), that can determine an end user's intent based on the user's utterances is a challenging task, due in part to the subtleties and ambiguities of natural language, the dimensions of the input space (e.g., possible user utterances), and the size of the output space (number of intents). In some bot systems, intents can be identified based on a user's utterances by training the bot with training utterances and associated intents. For example, in a shoe store bot system that may be interested in capturing a user intent to "buy shoes," the high-level intent of "buy shoes" may be fed into the user intent classification engine, which may then generate utterances such as "I want to buy shoes" and "I want a pair of shoes." Training phrases such as "I'm looking for shoes," "I want shoes," etc. can also be provided to train the user intent classification engine. In some embodiments, the user intent classification engine may expand the example phrases and attempt to match user utterances by using example phrases. For example, the user intent classification engine may match the phrase "I want to buy boots" with the intent "buy shoes" because a natural language processor (e.g., natural language processor 152) can understand that the semantic distance between boots, shoes, or slippers is short and they all refer to the same category of thing. Such a user intent classification engine may work for some chatbot interactions but cannot handle more complex tasks.

[0083] Instead of using intent matching as described above, some bot systems can identify user intent by combining low-level analysis results (syntactic analysis, entities, key phrases) with a knowledge graph for a specific domain. The knowledge graph (or ontology) may link intents, phrases, nouns, and adjectives to concepts within the domain. For example, a bot can use a natural language processor to parse user input and then use the analysis results to navigate through the knowledge graph to discover the user's intent at a semantic level. The bot can then use the user's intent to determine the most appropriate next response to the user.

[0084] A knowledge graph takes information and organizes it into an ontology, which contains a set of concepts and categories in a subject area or domain and describes their properties and the relationships between them. In some cases, a knowledge graph can also apply a reasoner to derive new knowledge. A knowledge graph captures facts about places, people, things, processes, applications, and the relationships between them, and can use such information to provide more relevant information to users performing searches or queries, for example. A knowledge graph typically represents a knowledge base using a directed graph with labeled edges (representing relationships) between nodes or vertices (representing entities). For example, a knowledge graph might represent a (subject, predicate, object) hierarchy such as (Obama, born in, Hawaii) or (Brazilian football team, has position, goalkeeper). The formula may be based on a set of triples of the form:

[0085] 3 illustrates an example knowledge graph 300. The knowledge graph 300 may include multiple vertices 310, which may represent, for example, places (e.g., Paris), people (e.g., Bob, Bill, and Leonardo da Vinci), things (e.g., the Eiffel Tower and the Mona Lisa), concepts (e.g., time, age / era, dimensions, and categories of things such as cities, people, and places), processes, applications, etc. The vertices 310 are connected by labeled edges 320, which, for each pair of connected vertices, describe the relationship between the two vertices. The edges 320 may be unidirectional or bidirectional. A vertex 310 can be obtained from another vertex 310 by traversing the knowledge graph 300 by following the paths formed by the edges 320.

[0086] 4A-4D illustrate an example process for building a knowledge graph based on a knowledge base such as a book or Wikipedia®. In FIG. 4A, a string 410 (e.g., "Leonardo da Vinci") can be obtained from the knowledge base. In FIG. 4B, an entity 420 (e.g., "Leonardo da Vinci") can be identified and recognized from the string 410. In FIG. 4C, an entity 432 related to the entity 420 can be identified from the knowledge base and added to the graph 430. In FIG. 4D, relationships between the entities 432 can be determined from the knowledge base and added to the graph as edges 442 or connections between the entities 432 to create a knowledge graph containing multiple entities 432 and edges 442. A digraph 440 can be formed. The entities 432 and edges 442 can be represented as triples of the form (subject, predicate, object), (entity, attribute, value), or other forms. Labels on the edges 442 can indicate relationships between the entities 432.

[0087] Knowledge graph 440 can be used as a seed graph for building a larger-scale knowledge graph. For example, knowledge graph 440 can be further expanded by identifying related entities from entities 432 in knowledge graph 440 and relationships between related entities and entities 432 in knowledge graph 440 from a knowledge base or a reference knowledge base (e.g., an external knowledge base) and adding them to knowledge graph 440. Expanding knowledge graph 440 can continue until, for example, knowledge in a given domain is exhausted, resulting in a knowledge graph for the given domain.

[0088] Applications of natural language understanding knowledge graphs include QnA, such as answering questions by embedding relevant sections of the knowledge graph, or generating relevant query subgraphs using reward functions trained by deep convolutional neural networks. Knowledge graphs can also be used, for example, by search engines to provide more relevant information in response to user searches, or by chatbots to perform intent classification.

[0089] Chatbots can perform better in specific domains with sufficient data. For a given bot, the user-provided dataset may contain, for example, only about 50-100 utterances. This limited dataset may be insufficient to build a knowledge graph that the bot can use to function properly. By using reference knowledge graphs or knowledge bases such as WikiData, Freebase, DBpedia, etc., knowledge within a specific knowledge domain can be expanded to generate a customized knowledge graph.

[0090] In some embodiments, a knowledge graph can be expanded, for example, by randomly walking through a reference knowledge graph or other knowledge base and selecting information from the reference knowledge graph or knowledge base, for example, based on how frequently nodes in the reference knowledge graph or knowledge base are visited. However, the quality of a knowledge graph generated by a random walk may be insufficient for applications such as chatbots. For example, a knowledge graph generated by a random walk may be noisy.

[0091] According to certain embodiments, a method is disclosed for building a customized knowledge graph for a given domain (e.g., a particular bot) using a customer dataset (e.g., including user utterances, intents, entities, and questions and answers (QnAs)) and a reference (e.g., external) knowledge base (e.g., WikiData). The customized knowledge graph may be domain-specific and may be generated through two main stages. First, entities may be extracted from the customer dataset, such as user-provided utterances, and used as vertices or nodes of an initial knowledge graph that can be used as a seed graph for expansion. The customized knowledge graph may be initialized by forming a seed graph using these extracted entities and the links between them. In a second stage, the seed graph is expanded by traversing the reference knowledge graph and selecting candidate entities to add to the customized knowledge graph using a finite state machine that can map entities in the seed graph to the reference knowledge graph as starting vertices for traversing the reference knowledge graph. The selected candidate entities from the reference knowledge graph can then be filtered based on their priority scores and added to the customized knowledge graph.

[0092] FIG. 5 is a simplified flowchart 500 illustrating an example method for building a customized knowledge graph (e.g., for a chatbot) according to certain embodiments. The process shown in FIG. 5 and described below is for illustrative purposes only and not limiting. While FIG. 5 depicts various operations occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, these operations may be performed in some different order, or some operations may be performed in parallel. Furthermore, embodiments of this method may be implemented in hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments for performing the associated tasks may be stored in a computer-readable medium, such as a storage medium. One or more processors may perform the associated tasks.

[0093] At 502, the computing system can receive a user dataset. The user dataset can include, for example, user utterances, user-defined entities and intents, user-provided questions and answers, etc. As previously mentioned, in some embodiments, the user dataset can include, for example, about 50 to about 100 utterances, each of which may or may not be tagged with an associated intent.

[0094] 504, part-of-speech tagging (see, for example, Kristina Toutanova et al., “Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network,” Proceedings of HLT-NAACL 2003, pp. 252-259), named entity recognition, etc. Named-entity recognition (NER) (see, for example, Jenny Rose Finkel et al., “Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling,” Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL 2005), pp. 563-370), constituency parsing using, for example, CoreNLP (see, for example, Manning et al., “The Stanford CoreNLP Natural Language Processing Toolkit,” Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Entities can be extracted from a user dataset, such as a user's utterances, using techniques such as the Dexter2 technology (see, e.g., Ceccarelli et al., "Dexter: an open source framework for entity linking," Proceedings of the sixth international workshop on Exploiting semantic annotations in information retrieval, ACM 2013, pp. 17-20).

[0095] At 506, relationships between the identified entities can be identified based on the user data set, for example, using the entity extraction techniques described above. The identified entities can be linked or connected using the relationships to generate a seed graph that includes the identified entities (as nodes or vertices in the seed graph) and the relationships between the identified entities (as edges between the vertices). The edges can be directed edges, which include the direction of the edge between the entities. The edges can be unidirectional or bidirectional. The seed graph can include a small number of vertices and can be a sparse graph that may not include all of the relationships between the entities. The seed graph can be , may contain one or more weakly connected components (WCCs), which generally refer to maximal subgraphs of a directed graph such that any node can be reached by starting from any other node and traversing the edges along undirected paths (i.e., ignoring the direction of the edges). All nodes in a weakly connected component may have an out-degree or in-degree of at least 1. In contrast, a strongly connected component may refer to a subgraph in which there is a directed path from every node to every other node.

[0096] At 508-516, the seed graph can be iteratively expanded by gradually adding entities and relationships to the seed graph using a simulated annealing optimization method (see, e.g., Nourani & Andresen, “A comparison of simulated annealing cooling strategies,” J. Phys. A: Math. Gen. 51. 1998, 8373-8385). At each iteration, a temperature value can be set or adjusted at 508. As described in more detail below, the temperature can be used to determine a priority score for candidate entities to add to the knowledge graph.

[0097] At 510, for each weakly connected component in the seed graph, each entity (i.e., vertex) in the weakly connected component can be mapped to a reference knowledge graph and used as a starting vertex in traversing the reference knowledge graph to identify up to M candidate entities using a finite state machine, as described in more detail below. The Resource Description Framework (RDF) of the identified candidate entities can be saved as entries in a priority queue.

[0098] At 512, entries in the priority queue may be scored using a scoring function that is a function of temperature. In some embodiments, the total number of candidate entities identified and added to the priority queue at 510 may be greater than a certain threshold number N. Techniques such as reservoir sampling (see, e.g., J. Vitter, "Random sampling with a reservoir," ACM Trans. on Mathematical Software 11(1):37-57, Mar. 1985) may be implemented to improve the scoring of the priority queue. Reducing the number of entries in the knowledge queue to N may reduce the time required to calculate the priority scores of the entries, which may be computationally more expensive than traversing the knowledge graph. As described in more detail below, a priority score may be calculated for each of the N candidate entities in the N entries and the corresponding path leading to the candidate entity.

[0099] At 514, from the maximum N entries in the priority queue, the K entries with the highest scores may be selected, and the seed graph may be expanded by adding the corresponding entities and relationships identified by the K entries to the seed graph.

[0100] If the temperature is determined to be above the threshold at 516, the operations at 508-514 may be performed again in the next iteration, during which the temperature (and therefore the scoring function) may change and the entities in the weakly connected components may be different because the seed graph has been modified using the top K entries in the priority queue. If the temperature is below the threshold, optimization of the customized knowledge graph may terminate, and the customized knowledge graph can be used for a user application associated with the user dataset, such as a particular bot or search engine.

[0101] Table 1 shows an example of pseudocode for implementing a method for generating a customized knowledge graph, such as the method described above with respect to FIG. 5. The pseudocode in Table 1 shows an example of a finite state machine used to traverse a reference knowledge graph. In Table 1, the first line of pseudocode relates to identifying entities and links between entities based on a user dataset (e.g., user utterances). The second line of pseudocode relates to creating a seed graph. The third line is used to set the initial state of a simulated annealing process for expanding the seed graph to generate a customized knowledge graph, including initializing the temperature and priority queue. Lines 4-23 relate to operations performed at each temperature of the simulated annealing process, and lines 5-15 describe using the finite state machine to traverse the reference knowledge graph to identify candidate entities and store the candidate entity's RDF triplet as an entry in the priority queue, as described in more detail below. Lines 16-17 relate to reducing the number of entries in the priority queue for priority score calculation if the number of entries in the priority queue is greater than a threshold number N. Lines 18-22 relate to calculating the priority scores of the entries in the priority queue and expanding the seed graph by using the top K entries with the highest priority scores. The temperature is updated in line 23.

[0102] [Table 1]

[0103] The method described above in connection with FIG. 3 and Table 1 generally involves (1) extracting entities from user-provided utterances and constructing a seed graph based on the extracted entities and links between them, and (2) expanding the seed graph by iteratively traversing a reference knowledge graph using a finite state machine, each of which This will be explained in detail below.

[0104] Entities can be extracted from a user dataset, such as a set of unprocessed sentences (e.g., an utterance), and used as vertices in a knowledge graph, and existing relationships between entities can be determined and used as edges between vertices in the knowledge graph. For example, a list of candidate entities can be generated by performing part-of-speech tagging, named entity recognition, and phrase structure analysis using CoreNLP techniques, as described above with respect to 504 in FIG. 5, on the utterance. Part-of-speech tagging techniques include labeling words with their appropriate parts of speech, such as nouns, verbs, adjectives, adverbs, pronouns, etc. Named entity recognition (also known as entity identification, entity chunking, or entity extraction) involves locating named entities in unstructured text and classifying them into predefined categories, such as names, organizations, places, medical codes, time expressions, quantities, monetary values, percentages, etc. Phrase structure analysis involves extracting a parse tree based on the phrase structure from the sentence to represent the syntactic structure of the sentence according to a phrase structure grammar. For example, a phrase-structure-based parse tree for the sentence "John hit the ball" may start with this sentence (S) and end with each of the leaf nodes (John, hit, the, ball). In this case, "John" is a noun (N) node. "hit" may be a verb (V) node (transitive in this example), "the ball" may be a noun phrase (NP) that is a branching node, "the" may be a determiner (D or DT) node, or "ball" may be a noun (N) node.

[0105] In some embodiments, the following preprocessing rules may be applied when generating the candidate entity list: (a) ensure that the NER is not null and ensure that words are part of a noun phrase (e.g., "apple tree" or "the ball") and tagged with the NER, (b) verify a match (extracted entity) if a proper noun (NNP, which designates a specific person, place, thing, or idea, e.g., "John") is dominated by the NP (i.e., the NP is the parent of the NNP in the semantic parse tree), (c) verify a match (extracted entity) if a common noun (NN, which designates a non-specific person, place, thing, or idea, e.g., "desk") is dominated by the NP (i.e., the NP is the parent of the NN in the semantic parse tree), (d) stripe personal pronouns (PRP, e.g., "I," "he," or "she") and determiners (DTs) from the above, (e) split the NER phrase into chunks, and (f) remove numbers and dates from the NER phrase. For example, applying the above rule to the sentence "How much did I spend from my checking account on ServiceA?" will result in a list of candidate entities that includes ["checking account", "ServiceA"]. In another example, applying the above rule to the sentence "I want to go to an outdoor cafe on Pioneer Day" will result in a list of candidate entities that includes ["Pioneer Day", "outdoor cafe", "outdoor cafe on Pioneer Day"].

[0106] In some embodiments, a list of candidate entities may be generated by detecting parallel mentions of entities in text using the Dexter2 technology described above. The two candidate entity lists may then be combined to generate a single candidate entity list. The combined candidate entity list may then be used to generate a seed graph for generating a knowledge graph.

[0107] In some embodiments, Dexter2 technology may be used to link candidate entities to corresponding entries in a reference knowledge graph such as WikiData, where Dexter2 technology may be used to link candidate entities to, for example, Wikipedia entries, and then map the Wikipedia entries to WikiData entries. For example, the entities (and corresponding In another example, entities (and their WikiData IDs) extracted from the example sentence "Michael Jordan played basketball at an outdoor cafe on Pioneer Day" might include "cafe" (Q30022), "Pioneer Day" (Q30023), and "checking account" (Q695430). Q254295), "basketball" (Q5372), "Michael Jordan" (Q41421), and "outdoor" (Q3312129). ,By using the Mediawiki Webservice API, it is possible to search for entities that do not match ,the above method.

[0108] Table 2 shows an example of a method for extracting entities from a user dataset (e.g., an utterance) and building a seed graph, such as the method described above. In operation 1 shown in Table 2, entities are extracted from raw sentences of the user utterance, for example, using part-of-speech tagging, named entity recognition, or phrase structure analysis using CoreNLP techniques described above.

[0109] [Table 2]

[0110] In operation 2, a candidate entity list can be generated by applying a set of preprocessing rules, as described above. The preprocessing rules may include, for example, (a) verifying that named entity recognition is not null and verifying that words are part of noun phrases, (b) verifying a match (extracted entity) if a proper noun is dominated by an NP (i.e., NP is the parent of NNP in the semantic parse tree), (c) verifying a match if a common noun is dominated by an NP (i.e., NP is the parent of NN in the semantic parse tree), (d) striping pronouns (PRPs) and determiners (DTs) from the above, (e) splitting NER phrases into chunks, and (f) removing numbers and dates from NER phrases. The final list after the above operations is a set of results for each individual rule.

[0111] In act 3, a graph lookup query can be determined based on the preprocessing rules described above, from the results of the parsing of the utterance in act 1, and a list of candidate entities generated by an entity linking technology such as the Dexter 2.0 framework described above. As used herein, entity linking may generally refer to matching an entity with the correct knowledge base entry when there is ambiguity. As an example, from the sentence "On July 20, 1969, the Apollo 11 astronauts Neil Armstrong, Michael Collins, and Edwin 'Buzz' Aldrin Jr. realized President Kennedy's dream," "Apollo 11" (space flight mission), "astronauts" "Neil Armstrong" (astronaut), "Michael Collins" (astronaut), "Edwin 'Buzz' Aldrin" (astronaut), and "John F. Kennedy" (astronaut). In another example, the search query "Michael Jordan" can be used to determine the search result for "Michael Jordan" (basketball player). and a graph lookup query for "Chicago Bulls" can be determined. Dexter2 technology can also be used to link graph lookup queries to corresponding entries in a reference knowledge base, such as a reference knowledge graph (e.g., WikiData). For example, Dexter2 technology can be used to link graph lookup queries to Wikipedia entries, which can then be mapped to WikiData entries.

[0112] In act 4, for each entity that does not match a reference knowledge base entry, a MediaWiki Webservice API can be called for entity linking and / or fetching information from the reference knowledge base, e.g., using Dexter 2.0. In some embodiments, in act 5, the returned WikiData entity IDs can be cached, e.g., using Redis, to reduce future API calls.

[0113] In operation 6, all possible combinations (N selects two combinations) of all entities in each utterance can be checked. If a link exists between two of the entities in the utterance, the link can be added to the seed graph. At the end of operation 6, a seed graph G=(V,E) may be generated, where vertex V contains the set of identified entities and edge E contains the link between any two of the entities. Checking for links between two entities can also be done for entities across the entire user dataset.

[0114] As mentioned previously, a user dataset may be small, such as approximately 50-100 utterances. Therefore, the seed graph may contain only a small number of entities, such as approximately 50-100 entities. Additionally, the links between entities in the seed graph may be sparse. The seed graph may need to be expanded to include additional information useful for a given domain, such as a particular chatbot. According to certain embodiments, a finite state machine may be used to traverse a reference knowledge graph, such as WikiData, to identify additional information (e.g., entities and links) that can be added to the seed graph to generate a customized knowledge graph. In some embodiments, a priority queue and a priority function may be used to determine which entities and / or links to add to the knowledge graph. In some embodiments, a simulated annealing process may be used to iteratively optimize the customized knowledge graph. An example method is described in the pseudocode shown in Table 3.

[0115] [Table 3]

[0116] In act 1 of the method shown in the pseudocode of Table 3, the simulated annealing process can be started by setting an initial temperature and initializing the priority queue, e.g., creating an empty priority queue.

[0117] In act 2(a), for each weakly connected component w of the seed graph G, g Each entity in can be mapped to a reference knowledge graph such as WikiData, and a finite state machine can be used to map the weakly connected components w gControlling the traversal of the reference graph from each entity in the For example, if the traversed vertex is a disambiguating vertex, the state machine can proceed to a disambiguating state, if the traversed vertex is a forbidden vertex, the state machine can proceed to a forbidden state, if the traversed vertex is an entity vertex, the state machine can proceed to an entity state, and the entity state can be g If the vertex has been visited M times for the specified Wikimedia project, the state machine may proceed to the "End" state. A disambiguating vertex, as used herein, refers to a knowledge graph entry or knowledge base entry that distinguishes between multiple entries that share the same or similar label. A forbidden vertex, as used herein, refers to a vertex in a knowledge graph (e.g., Wikidata) that either increases the sparseness of the customized knowledge graph or does not contribute meaningful information to the customized knowledge graph, and is therefore not suitable for addition to the customized graph. Forbidden vertices may include, for example, dates and / or times (e.g., time periods, points in time, etc.), links to other reference knowledge base identifiers (IMDb, Freebase, etc.), Wikimedia project, article, and list pages, pages outside the main Wikimedia knowledge tree, and duplicate and redirected entries.

[0118] 6 illustrates an example finite state machine (FSM) 600 for controlling the traversal of the reference knowledge graph in each iteration of a simulated annealing process, according to certain embodiments. In a start state 610 of the FSM 600, the traversal may begin at a vertex in the reference knowledge graph that is connected to a weakly connected component w in the seed graph G. g may correspond to entities in

[0119] If the next vertex connected to the starting vertex is a disambiguation vertex, the FSM 600 can transition to the disambiguation state 640. For example, the disambiguation state 640 can be reached whenever a disambiguation vertex in WikiData is reached. If the next vertex is not a disambiguation vertex but a forbidden vertex, the FSM 600 can transition to the forbidden state 630. An entry in the reference knowledge graph (e.g., WikiData) that is not a disambiguation vertex or a forbidden vertex is an entity w, and the FSM 600 can transition to the entity state 620. In this case, the resource description framework (RDF) for entity w can be added to a priority queue buffer (e.g., a cache). The RDF can be, for example, a triplet in the form of subject-predicate-object. The subject can refer to a resource, the object can refer to a property or aspect of the resource, and the predicate can refer to a relationship between the subject and the object.

[0120] All possible transitions between the entity state, disambiguation state, and forbidden state are allowed except for the transition from forbidden state 630 back to forbidden state 630, because it is undesirable to add any path to the knowledge graph through a forbidden vertex. Through this process, closely related entities in the user dataset (and seed graph) may be identified, and the graph search may be expanded using disambiguation vertices, ignoring forbidden vertices. The end state 650 is reached when the entity state 620 is moved to a weakly connected component w g may have been visited M times for each of the reference knowledge graphs, or when there are no other directly or indirectly linked vertices to traverse. The traversal of the reference knowledge graph in an iteration may stop at an end state 650.

[0121] In one example, the starting vertex has a link to the vertex "Obama" in the reference knowledge graph. If so, the FSM 600 can transition from the start state 610 to the entity state 620, adding the RDF for the vertex "Obama" to a priority queue buffer (e.g., pq_cache). The vertex "Obama" can have "America" ​​(linked by, say, the "president of" relationship) and "20 January 2009" (linked by, say, the "start time" relationship). If two vertices are linked, the vertex "20 January 2009" is Once traversed, FSM 600 can go to forbidden state 630 because the date and time vertex is a forbidden vertex. The next vertex to be traversed is vertex "America", which is a valid entity. Therefore, FSM 600 again goes to the entity state 630. The process can move to a priority queue state 620 and add the RDF for “America” to the priority queue buffer pq_cache. This process adds vertices to “America” and / or weakly connected components w of the seed graph G. g , can be repeated until the entity state 620 has been visited M times, or when there are no other directly or indirectly linked vertices to traverse.

[0122] In operation 2(b) of Table 3, a priority score may be calculated for each entry in the priority queue buffer using a scoring function that is a function of temperature. In some embodiments, if the total number of entities visited during an iteration exceeds a threshold N, the reservoir sampling technique described above with respect to 512 in FIG. 5 may be performed to reduce the number of entities in the priority queue buffer to N, thereby reducing the number of priority scores to be calculated. This may be done for time optimization purposes, since calculating priority scores may be more computationally expensive than traversing the knowledge graph using a finite state machine.

[0123] For example, after candidate entities are identified and corresponding RDF triples are stored in the priority queue buffer in act 2(a) in an iteration, a priority score can be calculated for each entity e in the priority queue and the corresponding path taken to reach entity e according to a priority function.

[0124]

number

[0125] where WCC(G') is used to identify weakly connected components in the graph. is the weakly connected component function used in (w, e), t is the current time step, T(t) is the temperature at time t, k, η and Z are real constants, and G' is the new graph formed by merging graph G with the set of edges between (w, e). The new graph G' can be expressed as follows:

[0126]

number

[0127] ρ(G) corresponds to the graph density and can be calculated as follows:

[0128]

number

[0129] where G is a graph with edges E and vertices V, |E| is the number of edges in graph G, and |V| is the number of vertices in graph G.

[0130] In operation 2(c) of Table 3, the entry in the priority queue with the highest priority score may be selected and added to the seed graph G. In operation 2(d), a new iteration of the processing of operation 2 may be initiated by updating the temperature of the simulated annealing process. The temperature at time step t may be determined, for example, as follows:

[0131]

number

[0132] A cooling schedule can be selected based on its effect on the priority function shown in equation (1). The priority function shown in equation (1) can prioritize adding new vertices when the temperature is high to increase the number of entities in weakly connected components. As the system cools, the priority function can place more weight on graph density to add more edges between vertices already present in the graph. The two parts of the priority function can be normalized by some factor (e.g., Z and η) so that they are similar in magnitude.

[0133] 7 illustrates an example seed graph 700 generated based on a user dataset (e.g., user utterances), such as those described above with respect to Table 2, according to certain embodiments. In this example, seed graph 700 describes the relationships between web applications and their parent companies. Seed graph 700 includes the following:<SoftwareA, part of, CompanyA> As shown, seed graph 700 may be a sparse graph with very few entities 710 and very few links 720 between entities 710. Links 720 are labeled with the relationships between entities 710. Figure 7 is intended solely to illustrate the sparseness of seed graph 700, and the specific content of seed graph 700 is not intended for ease of reading.

[0134] FIG. 8 shows an example of a customized knowledge graph 800 generated based on seed graph 700 and the techniques described above with respect to Table 3. FIG. 9 is an expanded view of a portion 810 of knowledge graph 800. FIGS. 8 and 9 are for illustrative purposes only, and the specific content of seed graph 800 is not intended for ease of reading. Compared to seed graph 700, knowledge graph 800 may have many more entities 820 and links 830 between entities 820. Knowledge graph 800 contains various information about web applications and surrounding concepts. Some examples include:<Internet Message Access Protocol, part of, email> ,<email filtering, instance of, anti-spam> ,<RunKeeper, instance of, mobile app> ,<RunKeeper, operating system, Android> , and<CompanyA, country, United States of America> The knowledge graph 800 also includes, for example,<alternative rock, subclass of, rock music> and<calendar, instance of, chronology> It may contain some information that may not be directly related to the domain of the dataset, such as: such information, possibly from other domains, allows the graph to be more easily generalized given unseen data.

[0135] Some examples of RDF triples (in the format "subject | predicate | object") that form Knowledge Graph 800 are "CompanyA Street View | developer | CompanyA", "email filtering | instance of | anti-spam", "BugMeNot | instance of | service on "internet", "CompanyB | country | United States of America", "CompanyC.com | software engine | ProductC", "RunKeeper | operating system | Android", "RunKeeper | instance of | mobile app", "ServiceA | instance of | online dating service", "Internet Message Access Protocol | part of | email", "software release life cycle | instance of | software engineering", "CompanyD | owned by | CompanyE", "CompanyF | parent organization | CompanyG", "G Suite | industry | email", "ProductA | publisher | CompanyH", "ProductA | developer | CompanyH may include, etc.

[0136] Table 4 shows statistical results for an example knowledge graph before and after augmentation with a reference knowledge graph. The statistics include, for example, the number of vertices, the number of edges, the graph density, and the number of weakly connected components in the dataset. As shown in Table 4, for datasets with a large number of starting vertices, the technique does not need to add many new vertices but only adds more edges. For SNIPS, the number of vertices and edges increases, but the graph density decreases because SNIPS is not a limited knowledge domain dataset. However, for smaller seed graphs, a significant number of new vertices can be added. This is because for larger datasets, entities identified for the seed graph may be very close in the reference knowledge graph. In addition, with a large number of starting vertices, while traversing the reference knowledge graph may not discover many new vertices, it is more likely to discover new relationships between these entities. On the other hand, for smaller datasets and therefore smaller seed graphs, the technique is more likely to discover many new vertices.

[0137] [Table 4]

[0138] Knowledge graphs can be used in many applications related to natural language understanding. For example, knowledge graph embedding (KGE) techniques are used to embed knowledge graphs into the By embedding the components of a knowledge graph, including entities and relationships, into a continuous vector space, the inherent structure of the knowledge graph can be preserved while simplifying manipulation. The entity and relationship embeddings can then be used, for example, for knowledge graph completion, relationship extraction, entity classification, entity resolution, question answering, etc.

[0139] In general, KGE techniques represent entities and relationships in a continuous vector space and then measure their plausibility by using a distance- or similarity-based scoring function for each fact and other information (e.g., entity type, relationship path, text description, or logic rule). The embeddings of entities and relationships can then be obtained by maximizing the overall plausibility of the observed facts. Knowledge graph embeddings may contain more semantic meaning compared to word vectors and can be used to augment feature representations based on word vectors. KGE, for example, can capture the semantic meaning of multi-word tokens (e.g., phrases). However, especially in small datasets with many unique and important words, we have been able to perform a classification on datasets using out-of-vocabulary (OOV) words, such as words without word vectors. By improving the predictive power of the proposed method, we can capture the semantic representation of entities beyond typical word vectors.

[0140] As mentioned above, a chatbot is a computer program or application capable of conducting a conversation through auditory or textual means by simulating the behavior of a human conversational partner. Chatbots can be used in dialog systems for a variety of practical purposes, including customer service or information retrieval. Some simpler chatbots can scan input for key words and then retrieve responses from a database with the best matching keywords or most similar wording patterns. Some chatbots can more closely simulate humans by using more advanced natural language processing techniques.

[0141] According to certain embodiments, knowledge graph embeddings can be used for intent classification, where a task may involve, for example, a chatbot determining what action a user wants to perform given an input natural language utterance. The chatbot can improve its performance in user intent classification by using both knowledge graph embeddings and word vectors extracted from, for example, a customized knowledge graph generated using the above method.

[0142] According to certain embodiments, embeddings for WikiData can be pre-trained using translational distance techniques, such as the TransE technique (see, e.g., Bordes et al., “Translating Embeddings for Modeling Multi-relational Data,” Advances in Neural Information Processing Systems 26. Curran Associates, Inc., 2013, pp. 2787-2795) and its extensions (see, e.g., Vrandecic & Krotzsch, “Wikidata: A Free Collaborative Knowledgebase,” Communications of the ACM 57:10, 2014, pp. 78-85), because translation-based modeling techniques are well suited for intent-specific applications. TransE is an energy-based model for learning low-dimensional embeddings of entities, where relationships are represented as translations in the embedding space. That is, if (h,r,t) holds (e.g. is proper RDF), then the embedding of tail entity t is close to the sum of the embedding of head entity h and a vector that depends on relation r (t ≈ h + r). Otherwise, the embedding of tail entity t will be very far from the sum of the embedding of head entity h and a vector that depends on relation r.

[0143] KGE techniques (e.g., translational distance or similarity-based KGE techniques) generally involve three operations: (1) representing entities and relationships, (2) defining a scoring function, and (3) learning the entity and relationship representations. The first operation specifies how the entities and relationships are represented in a continuous vector space. Entities are typically represented as vectors, such as deterministic points in a vector space. Relationships are generally interpreted as operations in a vector space that can be represented as vectors, matrices, tensors, multivariate Gaussian distributions, or mixtures of Gaussian functions. The second operation specifies how the scoring function f r Applying (h,t) to each fact (h,r,t) measures its plausibility. For example, in TransE technology, the scoring function can be defined as the negative distance between h+r and t.

[0144]

number

[0145] Observed facts in a knowledge graph generally have higher scores than unobserved facts. The third operation involves an optimization process that learns representations (e.g., embeddings) of entities and relationships by maximizing the overall plausibility of the observed facts.

[0146] In some embodiments, for all entities and corresponding relationships in graph G, the embeddings can be first initialized with pre-trained embeddings and then re-trained using a technique such as the TransE technique and a different scoring function. For example, the scoring function for the re-training process can be changed from equation (5) to the following equation:

[0147]

number

[0148] In the formula, θ r are the weights for the entities and relations that belong to the generated graph. This weighted scoring function is used in TransM (e.g., M. Fan et al., It is inspired by Transition-based knowledge graph embedding with relational mapping properties (see "Transition-based knowledge graph embedding with relational mapping properties," in Proc. 28th Pacific Asia Conf. Language Inf. Comput., 2014, pp. 328-337), and gives more weight to entities and relations in the graph G. A max-margin hinge loss function can be used following TransE's open-world training paradigm. The embedding can be updated with respect to the following gradient:

[0149]

number

[0150] where ||h||2 = 1 and ||t||2 = 1. Equation (7) may include a max-margin hinge loss function, where the hyperparameter γ is the margin (> 0). To prevent the embedding of entities and relationships in graph G from becoming too sparse, an additional regularization term is included in equation (7), where λ is the regularization strength (> 0). T and T' denote the training triples and corrupted triples in each mini-batch, respectively. This is modeled as a transfer learning process, where retraining the graph embedding can help fine-tune the vectors of the generated knowledge graph by giving more weight to entities and relationships present in the knowledge graph. This process can reduce noise in the embedding and compensate for TransE's weakness in modeling non-one-to-one relationships by prioritizing components of the generated graph that are less likely to have such non-one-to-one relationships.

[0151] Table 5 shows an example of how to use the above knowledge graph embeddings for, e.g., intent classification.

[0152] [Table 5]

[0153] In the example shown in Table 5, in act 1, embeddings can be pre-trained, e.g., on Wikidata. In act 2, embeddings can be initialized for all entities and corresponding relationships in the knowledge graph G using the pre-trained embeddings obtained in act 1, and then re-trained using techniques such as TransE and a different, weighted scoring function. In act 3, for a given utterance, all entities in the utterance can be extracted using the entity extraction and linking process described above for Table 2. Next, all entity embeddings (computed above) for entities in the sentence are averaged. In addition, the sentence can be cleaned in parallel (e.g., trimming, removing vocabulary words, removing numbers and unknown characters, etc.) to reduce sparseness, and GloVe vectors (see, e.g., Pennington et al., “Glove: Global vectors for word representation,” Proceedings of the Empirical Methods in Natural Language Processing (EMNLP), 2014) can be averaged for the remaining words in the sentence. Next Then, the averaged GloVe vector and averaged knowledge graph embedding for this sentence can be concatenated, and principal component analysis (PCA) can be performed on the concatenated vector, as shown, for example, in Goikoetxea et al., “Single or multiple? combining word representations independently learned from text and wordnet,” Proceedings of AAAI, 2016. In operation 4, after PCA, the concatenated vector can be sent to a classifier for intent classification.

[0154] Table 6 shows the training and query times using the knowledge graph embedding and average GloVe vector techniques described above in relation to Table 5. KGE training times are for the first run without caching; subsequent runs will be faster.

[0155] [Table 6]

[0156] Additionally, an ablation study is used to determine which aspects of the overall process contribute to increased accuracy and F1 score. Experiments were conducted on five variants of the technique. In Variation Var1, no hyperparameter optimization (i.e., grid search) is performed, but rather the hyperparameters are randomly selected within a range. In Variation Var2, the knowledge graph is initialized using only vertices and edges (relationships) between entities discovered in the input utterance. In Variation Var3, a one-hop random walk traversal of a reference knowledge graph of maximum depth 5 is used, rather than the finite state machine described in this disclosure. In Variation Var4, feature vectors are generated by averaging GloVe vectors for each word in the input utterance. In Variation Var5, retraining of the knowledge graph embeddings as described with respect to Table 5 is not performed, but instead TransE embeddings pretrained on WikiData are used. The F1 score and accuracy results for these five variants are shown in Table 7 and compared to the results obtained without applying any of these variants.

[0157] [Table 7]

[0158] The results of the ablation study indicate that although hyperparameter optimization may result in small improvements in accuracy (as shown by comparing Var1 with the full algorithm in Table 7), this is insignificant compared to the contributions of other techniques disclosed herein. The performance degradation in the cases of random walk traversal (Var3 variant) and average GloVe vector (Var4 variant) indicates that knowledge graph embedding can capture more relevant information in the classification process. In particular, the F1 score drops significantly more than the accuracy. This may indicate that more semantic information about the utterance may be present in the feature vector, resulting in fewer false positives and false negatives. How to Initialize a Knowledge Graph Both the method of traversing the reference knowledge graph have a significant impact on the results. Retraining the knowledge graph embeddings can also improve performance, indicating that retraining the embeddings to prioritize components of the generated knowledge graph can be useful for modeling entities and relationships.

[0159] The techniques described herein may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores), hardware, or a combination thereof. The software may be stored in memory (e.g., on a non-transitory computer-readable storage medium such as a memory device). The techniques described herein may be implemented in a variety of different environments, including cloud environments (e.g., various clouds, including private, public, and hybrid cloud environments), on-premise environments, hybrid environments, etc.

[0160] 10 shows a simplified diagram of a distributed system 1000 for implementing an embodiment. In the illustrated embodiment, the distributed system 1000 includes one or more client computing devices 1002, 1004, 1006, and 1008 coupled to one or more communication networks 1010 via a server 1012. The client computing devices 1002, 1004, 1006, and 1008 may be configured to run one or more applications.

[0161] In various embodiments, server 1012 may be adapted to execute or provide one or more services or software applications that automatically generate relationships or mappings between user interface components of an application and data objects using request information about the application. For example, in particular embodiments, server 1012 may receive request information about an application, which may include a description of the application's desired functionality and may also include an image of the application's desired GUI. Server 1012 can then use the request information to generate mappings between the application's user interface components and one or more data objects.

[0162] In particular embodiments, the server 1012 may also provide other services or software applications that may be provided in a non-virtual or virtual environment. In some embodiments, these services may be provided through software as a service (Software as a Service). As a web-based or cloud service, such as a SaaS model The client applications may be provided to users of client computing devices 1002, 1004, 1006, and / or 1008. Users operating client computing devices 1002, 1004, 1006, and / or 1008 may then utilize one or more client applications to interact with server 1012 and utilize the services provided by these components.

[0163] In the configuration shown in Figure 10, server 1012 may include one or more components 1018, 1020, and 1022 that implement the functions performed by server 1012. These components may include software components executable by one or more processors, hardware components, or a combination thereof. It should be understood that a wide variety of system configurations are possible that may differ from distributed system 1000. Thus, the embodiment shown in Figure 10 is an example of a distributed system for implementing the system of the embodiment and is not intended to be limiting.

[0164] In accordance with the teachings of this disclosure, a user can use client computing devices 1002, 1004, 1006, and / or 1008 to provide requested information to server 1012 and receive information (e.g., mapping information) from server 1012. The client devices may provide an interface that allows a user of the client device to interact with the client device. The client devices also output information to the user via this interface. Although FIG. 10 shows only four client computing devices, any number of client computing devices may be supported.

[0165] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computing devices may include various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux-like operating systems, various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android®, BlackBerry®, Google Chrome® OS, including Palm OS®). Portable handheld devices may include cellular phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), personal digital assistants (PDAs), etc. Wearable devices may include Google Glass® head-mounted displays and other devices. The gaming systems may include various handheld gaming devices, Internet-connectable gaming devices (e.g., Microsoft Xbox® gaming consoles with Kinect® gesture input devices, Sony PlayStation® systems, various gaming systems offered by Nintendo®, etc.). The client devices may be capable of running a wide variety of applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications), and may use a variety of communication protocols.

[0166] Communications network 1010 may be any type of network known to those skilled in the art that is capable of supporting data communications using any of a variety of available protocols, including, but not limited to, TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), AppleTalk®, etc. By way of example only, communications network 1010 may include a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a wireless network operating under any of the Institute of Electrical and Electronics Engineers (IEEE) 1002.11 protocol suite, Bluetooth®, and / or any other wireless protocol), and / or any combination of these and / or other networks.

[0167] Server 1012 may be one or more general-purpose computers, dedicated server computers (for example, The servers 1012 may be configured as servers (e.g., personal computer (PC) servers, UNIX servers, midrange servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or other suitable configurations and / or combinations. The servers 1012 may include one or more virtual machines running a virtual operating system or other computing architecture involving virtualization, such as one or more flexible tools of logical storage that can be virtualized to maintain virtual storage for the servers. In various embodiments, the servers 1012 may be adapted to run one or more services or software applications that provide the functionality described in the above disclosure.

[0168] The computing systems within server 1012 may run one or more operating systems, including any of the operating systems described above, as well as commercially available server operating systems. Server 1012 may also run any of a variety of other server and / or middle-tier applications, including an HTTP (Hypertext Transfer Protocol) server, an FTP (File Transfer Protocol) server, a CGI (Common Gateway Interface) server, a JAVA server, a database server, etc. Exemplary database servers include commercially available operating systems from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), etc. This includes, but is not limited to, those available.

[0169] In some implementations, server 1012 may include one or more applications for parsing and consolidating data feeds and / or event updates received from users of client computing devices 1002, 1004, 1006, and 1008. By way of example, the data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third-party sources and continuous data streams that may include real-time events related to sensor data applications, financial stock tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. Server 1012 may also include one or more applications for displaying the data feeds and / or real-time events via one or more display devices of client computing devices 1002, 1004, 1006, and 1008.

[0170] The distributed system 1000 may also include one or more data repositories 1014, 1016. In particular embodiments, these data repositories may be used to store data and other information. For example, one or more of the data repositories 1014, 1016 may be used to store information such as requirements information, mapping information, and other information for one or more applications used or generated by the server 1012. The data repositories 1014, 1016 may be in a variety of locations. For example, a data repository used by the server 1012 may be local to the server 1012 or may be remote from the server 1012 and communicate with the server 1012 via a network-based or dedicated connection. The data repositories 1014, 1016 may be of different types. In particular embodiments, the data repository used by the server 1012 may be a database, for example, a relational database such as a database provided by Oracle Corporation®. One or more of these databases may be adapted to allow data to be stored, updated, and retrieved from the database in response to SQL-formatted commands. The data repositories used by the applications may be , may be of various types, such as a key-value store repository, an object store repository, or a general-purpose storage repository backed by a file system.

[0171] In certain embodiments, features described in this disclosure may be provided as services via a cloud environment. FIG. 11 is a simplified block diagram of a cloud-based system environment that may provide various services as cloud services, according to certain embodiments. In the embodiment shown in FIG. 11, a cloud infrastructure system 1102 may provide one or more cloud services that users may request using one or more client devices 1104, 1106, and 1108. The cloud infrastructure system 1102 may include one or more computers and / or servers, which may include those described above with respect to server 1012. The computers in the cloud infrastructure system 1102 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0172] The network 1110 can facilitate communication and exchange of data between the client devices 1104, 1106, and 1108 and the cloud infrastructure system 1102. The network 1110 can include one or more networks. The networks can be of the same type or different types. The network 1110 can support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0173] The embodiment shown in Figure 11 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in other embodiments, cloud infrastructure system 1102 may have more or fewer components than those shown in Figure 11, may combine two or more components, or may have a different configuration or arrangement of components. For example, while Figure 11 shows three client computing devices, any number of client computing devices may be supported in alternative embodiments.

[0174] The term cloud service is generally used to mean services made available to users on demand via a communications network such as the Internet by a service provider's system (e.g., cloud infrastructure system 1102). Typically, in a public cloud environment, the servers and systems that comprise the cloud service provider's system are distinct from the customer's own on-premise servers and systems. The cloud service provider's systems are managed by the cloud service provider. Therefore, customers can use cloud services offered by the cloud service provider without purchasing separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's system can host applications, and users can order and use the applications on demand via the Internet without having to purchase infrastructure resources to run the applications. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, several cloud services, such as middleware services, database services, Java cloud services, and others, are offered by Oracle Corporation® of Redwood Shores, California.

[0175] In particular embodiments, cloud infrastructure system 1102 may provide one or more cloud services using different models, such as a Software as a Service (SaaS) model, a Platform as a Service (PaaS) model, an Infrastructure as a Service (IaaS) model, etc., including hybrid service models. Cloud infrastructure system 1102 may include a suite of applications, middleware, databases, and other resources that enable the provision of various cloud services.

[0176] The SaaS model allows applications or software to be delivered to customers as a service over a communications network such as the Internet, without the customer having to purchase the underlying application hardware or software. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by cloud infrastructure system 1102. Examples of SaaS services offered by Oracle Corporation® include, but are not limited to, various services for human resource / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, and others. In certain embodiments, the ability to automatically generate functional applications from application requirements information can be provided as a service based on the SaaS model.

[0177] The IaaS model is commonly used to provide flexible computing and storage capabilities by providing infrastructure resources (e.g., servers, storage, hardware, and networking resources) to customers as cloud services. Various IaaS services are offered by Oracle Corporation (registered trademark).

[0178] The PaaS model is typically used to provide platform and environment resources as a service that enable customers to develop, run, and manage applications and services without having to procure, build, or manage the environment resources. Examples of PaaS services provided by Oracle Corporation (registered trademark) include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, various application development solution services, and others.

[0179] Cloud services are typically provided on an on-demand, self-service basis, on a subscription basis, and in a flexible, scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services offered by cloud infrastructure system 1102 via a subscription order. Cloud infrastructure system 1102 then performs processing to provide the services requested in the customer's subscription order. For example, a user may provide request information for an application to the cloud infrastructure system and request the cloud infrastructure system to automatically generate a functional application from the request information. This may include automatically generating mappings between user interface components and data objects of the application, as described above. Cloud infrastructure system 1102 may be configured to provide one or more cloud services.

[0180] The cloud infrastructure system 1102 supports different deployment models. In a public cloud model, the cloud infrastructure system 1102 may be owned by a third-party cloud service provider, and cloud services may be offered to general public customers. These customers may be individuals or businesses. In certain other embodiments, in a private cloud model, the cloud infrastructure system 1102 may function within an organization (e.g., within a corporate organization), and services may be offered to customers within the organization. For example, these customers may be various departments, such as human resources, payroll, etc., or individuals within the company. In certain other embodiments, in a community cloud model, the cloud infrastructure system 1102 and the services offered may be shared among several organizations within an associated community. Various other models, such as hybrids of the above models, may also be used.

[0181] Client devices 1104, 1106, and 1108 may be of different types (e.g., client devices 1104, 1106, and 1108 shown in FIG. 11 ) and may be capable of operating one or more client applications. Users may use the client devices to interact with cloud infrastructure system 1102, such as requesting services provided by cloud infrastructure system 1102. For example, users may use the client devices to provide request information about an application to cloud infrastructure system 1102 and request the cloud infrastructure system to automatically generate a functional application from the request information. This includes automatically generating mappings between user interface components and data objects of the application, as described in this disclosure.

[0182] In some embodiments, the processing performed by cloud infrastructure system 1102 to provide the requested service may include big data analytics. This analytics may involve using, analyzing, and manipulating large data sets to detect and visualize various trends, behaviors, relationships, etc. within this data. This analytics may be performed by one or more processors, possibly processing the data in parallel, running simulations with the data, etc. The data used in this analytics may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0183] 11 , cloud infrastructure system 1102 may include infrastructure resources 1130 utilized to facilitate the provision of various cloud services offered by cloud infrastructure system 1102. Infrastructure resources 1130 may include, for example, processing resources, storage or memory resources, networking resources, etc.

[0184] In particular embodiments, to facilitate efficient provisioning of these resources to support the various cloud services offered by cloud infrastructure system 1102 to different customers, resources may be organized into resource sets or resource modules (also referred to as "pods"). Each resource module or pod may include a pre-integrated, optimized combination of one or more types of resources. In particular embodiments, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provisioned for database services, and a second set of pods may be provisioned for Java services, etc., which may include a different combination of resources than the pods in the first set of pods. For some services, provisioning these services may require a The services may share resources allocated to

[0185] Cloud infrastructure system 1102 itself may use services 1132 internally that are shared by different components of cloud infrastructure system 1102 and that facilitate the provisioning of services by cloud infrastructure system 1102. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelist services, high availability backup and recovery services, services enabling cloud support, email services, notification services, file transfer services, etc.

[0186] Cloud infrastructure system 1102 may include multiple subsystems. These subsystems may be implemented in software, hardware, or a combination thereof. As shown in FIG. 11 , the subsystems may include a user interface subsystem 1112 that allows users or customers of cloud infrastructure system 1102 to interact with cloud infrastructure system 1102. User interface subsystem 1112 may include a variety of different interfaces, such as a web interface 1114, an online store interface 1116 through which cloud services offered by cloud infrastructure system 1102 are advertised and available for purchase by consumers, and other interfaces 1118. For example, a customer may use a client device to request one or more services offered by cloud infrastructure system 1102 by placing a subscription order (service request 1134) using one or more of interfaces 1114, 1116, and 1118. These interfaces may include, for example, an online store user interface 1116 through which a customer can access and browse an online store that identifies cloud services offered by cloud infrastructure system 1102 and place a subscription order for one or more of the offered services to which the customer wishes to subscribe. Other interfaces may include a web interface 1114 and other interfaces 1118.

[0187] In particular embodiments, a service or subscription request may include information identifying the requesting user or customer and one or more services for which the customer wishes to subscribe. For example, a customer may place a subscription order for an automated application generation service offered by cloud infrastructure system 1102.

[0188] 11, cloud infrastructure system 1102 may include an order management subsystem (OMS) 1102 configured to process new subscription orders. As part of this processing, OMS 1120 may be configured to create an account for the customer if not already created, receive billing and / or account information from the customer to use to charge the customer for providing the requested services, verify the customer information, and, once verified, reserve the order for the customer and prepare the order for provisioning by coordinating various workflows.

[0189] Upon proper validation, the OMS 1120 can invoke an Order Provisioning Subsystem (OPS) 1124 configured to provision resources for this order, including processing, memory, and networking resources. Provisioning involves allocating resources for the order and providing the customer order. and configuring resources to facilitate the service requested by the customer. The manner in which resources are provisioned for an order and the type of resources provisioned may depend on the type of cloud service ordered by the customer. For example, according to one workflow, OPS 1124 may be configured to determine the specific cloud service being requested and identify the number of pods that will be pre-configured for that specific cloud service. The number of pods allocated for an order may depend on the size / amount / level / scope of the service requested. For example, the number of pods to allocate may be determined based on the number of users the service is to support, the duration for which the service is requested, etc. The allocated pods may then be customized to the particular requesting customer to provide the requested service.

[0190] Cloud infrastructure system 1102 may send a response or notification 1144 to the requesting customer to indicate when the requested service will be available for use. In some examples, information (e.g., a link) may be sent to the customer that enables the customer to begin using and avail themselves of the benefits of the requested service.

[0191] Cloud infrastructure system 1102 may provide services to multiple customers. For each customer, cloud infrastructure system 1102 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the orders, and providing the requested services to the customer. Cloud infrastructure system 1102 may also collect usage statistics regarding the customer's use of the subscribed services. For example, statistics may be collected about the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and downtime. This usage information may be used to bill the customer. Billing may be on a monthly basis, for example.

[0192] Cloud infrastructure system 1102 may provide services to multiple customers concurrently. Cloud infrastructure system 1102 may store information about these customers, possibly including copyright information. In particular embodiments, cloud infrastructure system 1102 includes an identity management subsystem (IMS) 1128 configured to manage customer information and provide separation of managed information so that information about one customer is not accessed from or commingled with information about another customer. IMS 1128 may be configured to provide various security-related services, such as identity services, e.g., information access management, authentication and authorization services, services for managing customer identities and roles and associated capabilities, etc.

[0193] 12 illustrates an exemplary computer system 1200 that may be used to implement certain embodiments. For example, in some embodiments, computer system 1200 may be used to implement any of the various servers and computer systems described above, such as model generation system 102 shown in FIG. 1. As shown in FIG. 12, computer system 1200 includes various subsystems, including a processing subsystem 1204 that communicates with multiple other subsystems via a bus subsystem 1202. These other subsystems may include a processing acceleration unit 1206, an I / O subsystem 1208, a storage subsystem 1218, and a communication subsystem 1224. Storage subsystem 1218 may include non-transitory computer-readable storage media, including a storage medium 1222 and a system memory 1210.

[0194] Bus subsystem 1202 provides a mechanism for allowing the various components and subsystems of computer system 1200 to communicate with each other as intended. Although bus subsystem 1202 is shown schematically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1202 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, a local bus, etc., using any of a variety of bus architectures. For example, such architectures include the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA (EISA) bus, the Video Electronics Standards Association (VESA) local bus, and the IEEE P138 bus. This may include a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured in accordance with the 6.1 standard.

[0195] The processing subsystem 1204 controls the operation of the computer system 1200 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processors may include single-core or multi-core processors. The processing resources of the computer system 1200 may be organized into one or more processing units 1232, 1234, etc. The processing units may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some embodiments, the processing subsystem 1204 may include one or more dedicated coprocessors, such as a graphics processor, a digital signal processor (DSP), etc. In some embodiments, some or all of the processing units of the processing subsystem 1204 may be implemented using customized circuitry, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0196] In some embodiments, the processing units within the processing subsystem 1204 can execute instructions stored in the system memory 1210 or the computer-readable storage medium 1222. In various embodiments, the processing units can execute various program or code instructions and maintain multiple programs or processes running simultaneously. At any given time, some or all of the program code to be executed may reside in the system memory 1210 and / or the computer-readable storage medium 1222, potentially including one or more storage devices. Through appropriate programming, the processing subsystem 1204 can provide the various functions described above. In examples where the computer system 1200 is running one or more virtual machines, each virtual machine may be assigned to one or more processing units.

[0197] In particular embodiments, a processing acceleration unit 1206 may optionally be provided to accelerate the overall processing performed by the computer system 1200, to perform customized processing, or to offload portions of the processing performed by the processing subsystem 1204.

[0198] I / O subsystem 1208 can include devices and mechanisms for inputting information into computer system 1200 and / or outputting information from or through computer system 1200. In general, use of the term "input device" is intended to include all conceivable types of devices and mechanisms for inputting information into computer system 1200. User interface input devices include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as a Microsoft Kinect® motion sensor, a Microsoft Xbox® 360 game controller, which allow a user to control and interact with the input device, and devices that provide an interface for receiving input using gestures and voice commands. User interface input devices may also include eye gesture recognition devices, such as a Google Glass® blink detector, which detects eye movements from a user (e.g., "blinks" while taking a picture and / or making a menu selection) and translates the eye gestures as input to an input device (e.g., Google Glass®). User interface input devices may also include voice recognition sensing devices, which allow a user to interact with a voice recognition system (e.g., Siri® navigator) via voice commands.

[0199] Other examples of user interface input devices may include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads, and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser range finders, and eye-tracking devices. User interface input devices may also include medical imaging input devices, such as computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices, such as MIDI keyboards, digital musical instruments, and the like.

[0200] In general, the use of the term output device(s) is intended to encompass all conceivable types of devices and mechanisms for outputting information from computer system 1200 to a user or to another computer. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems may be flat-panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, projection devices, touch screens, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / visual information, such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, audio output devices, and modems.

[0201] The storage subsystem 1218 provides a repository or data store for storing information and data used by the computer system 1200. The storage subsystem 1218 provides a tangible, non-transitory, computer-readable storage medium for storing the basic programming and data constructs that provide the functionality of some embodiments. Software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 1204, provide the functionality described above may be stored in the storage subsystem 1218. The software may be executed by one or more processing units of the processing subsystem 1204. The storage subsystem 1218 may also provide a repository for storing data used in accordance with the teachings of the present disclosure.

[0202] The storage subsystem 1218 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. As shown in FIG. 12, the storage subsystem 1218 includes a system memory 1210 and a computer-readable storage medium 1222. The system memory 1210 may include several memories, including volatile primary random access memory (RAM) for storing instructions and data during program execution, and nonvolatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some implementations, a basic input / output system (BIOS), containing the basic routines that help to transfer information between elements within the computer system 1200, such as during start-up, is typically stored in ROM. Typically, RAM contains data and / or program modules currently being operated on and executed by the processing subsystem 1204. In some implementations, the system memory 1210 may include several different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0203] 12, system memory 1210 may load running application programs 1212, program data 1214, and operating system 1216, which may include various applications such as a web browser, a middle-tier application, a relational database management system (RDBMS), etc. By way of example, operating system 1216 may include Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or various versions of mobile operating systems such as iOS®, Windows® Phone, Android® OS, BlackBerry® OS, Palm® OS operating systems, etc.

[0204] The computer-readable storage medium 1222 may store programming and data structures that provide functionality of some embodiments. The computer-readable storage medium 1222 may provide storage of computer-readable instructions, data structures, program modules, and other data for the computer system 1200. Software (programs, code modules, instructions) that, when executed by the processing subsystem 1204, provide the above-described functionality may be stored in the storage subsystem 1218. By way of example, the computer-readable storage medium 1222 may include non-volatile memory such as a hard disk drive, a magnetic disk drive, a CD-ROM, a DVD, an optical disk drive such as a Blu-Ray® disk, or other optical media. The computer-readable storage medium 1222 may include, but is not limited to, a Zip® drive, a flash memory card, a Universal Serial Bus (USB) flash drive, a Secure Digital (SD) card, a DVD disc, a digital video tape, etc. The computer-readable storage medium 1222 may also include solid-state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, solid-state ROM, etc., SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs.

[0205] In particular embodiments, storage subsystem 1218 may also include a computer-readable storage medium reader 1220 that may be further connected to a computer-readable storage medium 1222. Computer-readable storage medium reader 1220 may be configured to receive and read data from a memory device such as a disk, flash drive, or the like.

[0206] In particular embodiments, computer system 1200 may support virtualization techniques, including, but not limited to, virtualization of processing and memory resources. For example, computer system 1200 may provide support for running one or more virtual machines. In particular embodiments, computer system 1200 may execute a program such as a hypervisor to facilitate configuration and management of virtual machines. Each virtual machine may be assigned memory, computational (e.g., processors, cores), I / O, and networking resources. Each virtual machine typically runs independently from other virtual machines. A virtual machine typically runs its own operating system, which may be the same or different from the operating systems run by other virtual machines executed by computer system 1200. Thus, potentially multiple operating systems may be running simultaneously by computer system 1200.

[0207] The communications subsystem 1224 provides an interface to other computer systems and networks. The communications subsystem 1224 acts as an interface for sending and receiving data between other systems and the computer system 1200. For example, the communications subsystem 1224 may enable the computer system 1200 to establish a communications channel over the Internet to one or more client devices to send information to or receive information from the one or more client devices.

[0208] The communications subsystem 1224 can support both wired and / or wireless communication protocols. For example, in an embodiment, the communications subsystem 1224 can include a radio frequency (RF) transceiver component for accessing wireless voice and / or data networks (e.g., using cellular telephony, advanced data network technologies such as 3G, 4G, or EDGE (High Data Rates for Global Evolution), WiFi (IEEE 802.XX family of standards, or other mobile communications technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components. In some embodiments, the communications subsystem 1224 can provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0209] The communications subsystem 1224 can receive and transmit data in a variety of formats. For example, in some embodiments, the communications subsystem 1224 can receive incoming communications in the form of structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc. For example, the communications subsystem 1224 may be configured to receive (or transmit) data feeds 1226 in real time from users of social media networks and / or other communications services, such as web feeds such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party sources.

[0210] In particular embodiments, the communications subsystem 1224 may be configured to receive data in the form of a continuous data stream, which may include an event stream 1228 of real-time events and / or event updates 1230 that may be continuous or infinite in nature without a clear end. Examples of applications that generate continuous data include, for example, sensor data applications, financial stock ticker boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, and automotive traffic management applications. Examples of such measures include monitoring.

[0211] Communications subsystem 1224 may be configured to communicate data from computer system 1200 to other computer systems or networks. This data may be communicated in a variety of different formats, such as structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc., to one or more databases that may be in communication with one or more streaming data source computers coupled to computer system 1200.

[0212] Computer system 1200 may be one of a variety of types, including a handheld portable device (e.g., an iPhone® cellular phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a personal computer, a workstation, a mainframe, a kiosk, a server rack, or other data processing system. Due to the ever-changing nature of computers and networks, the description of computer system 1200 shown in FIG. 12 is intended merely as a specific example. Many other configurations are possible, having more or fewer components than the system shown in FIG. 12. Based on the disclosure and teachings herein, one of ordinary skill in the art will appreciate other aspects and / or methods for implementing various embodiments.

[0213] While specific embodiments have been described, various modifications, variations, alternative constructions, and equivalents are possible. The embodiments are not limited to operation in a particular data processing environment, but may freely operate in multiple data processing environments. Furthermore, while the embodiments have been described using a particular sequence of transactions and steps, it should be apparent to those skilled in the art that this is not intended to be limiting. While some flowcharts describe operations as a sequential process, many of these operations can be performed in parallel or simultaneously. Additionally, the order of operations may be re-specified. A process may have additional steps not included in the figures. Various features and aspects of the above-described embodiments may be used individually or together.

[0214] Additionally, while particular embodiments have been described using particular combinations of hardware and software, it should be understood that other combinations of hardware and software are possible. Particular embodiments may be implemented exclusively in hardware, exclusively in software, or using a combination thereof. The various processes described herein may be implemented on the same processor or any combination of different processors.

[0215] Where a device, system, component, or module is described as being configured to perform a particular operation or function, such configuration may be achieved by, for example, designing an electronic circuit to perform the operation, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, such as by executing computer instructions or code, or a processor or core programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes may communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, and the same pair of processes may use different techniques at different times.

[0216] In this disclosure, specific details are set forth to provide a thorough understanding of the embodiments. However, the embodiments may be practiced without these specific details. For example, well-known circuits, processes, algorithms, configurations, and techniques are shown without unnecessary detail so as not to obscure the embodiments. This specification provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of other embodiments. Rather, the above description of the embodiments provides one skilled in the art with an enabling description for implementing various embodiments. Various changes may be made within the function and configuration of the elements.

[0217] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will be apparent, however, that additions, subtractions, deletions, and other modifications and alterations may be made thereto without departing from the broader spirit and scope as set forth in the claims. Thus, while specific embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are intended to be within the scope of the following claims.

Claims

1. 1. A computer-implemented method for generating a customized knowledge graph for an application, the computer-implemented method comprising: receiving a user data set for the application; extracting entities from the user dataset; identifying links between the entities based on the user data set; creating a seed graph representing the entities and the links between them; identifying weakly connected components in the seed graph; For each weakly connected component in the seed graph, mapping entities in the weakly connected components to vertices in a reference knowledge graph; Identifying up to a first threshold number of entities in the reference knowledge graph by traversing the reference knowledge graph from the mapped vertices in the reference knowledge graph based on a finite state machine; storing a Resource Description Framework (RDF) of the identified entity as an entry in a buffer; calculating a priority score for the entry in the buffer; selecting a first set of entries from the entries in the buffer having the highest priority scores; and generating a version of the customized knowledge graph by adding entities and links identified by the first set of entries to the seed graph.

2. The computer-implemented method of claim 1 , wherein the user data set comprises a plurality of user utterances.

3. The steps of extracting entities and identifying links between entities include performing the following steps on the user data set: Part-of-speech tagging, Named Entity Recognition, or Phrase structure analysis 10. The computer-implemented method of claim 1, comprising performing at least one of:

4. 2. The computer-implemented method of claim 1 , wherein mapping entities in the weakly connected components to vertices in a reference knowledge graph comprises mapping the entities to vertices in the reference knowledge graph using Dexter2.

5. the finite state machine includes a disambiguation state, an entity state, a prohibition state, and a termination state; The step of traversing the reference knowledge graph includes: entering the disambiguation state when the next vertex in the reference knowledge graph is a disambiguation vertex; When the next vertex in the reference knowledge graph is a forbidden vertex, if the current state of the finite state machine is not the forbidden state, entering the forbidden state; When the next vertex in the reference knowledge graph is an entity vertex, Entering the entity state, saving the RDF of the next vertex in the reference knowledge graph as an entry in the buffer; and entering the termination state when the number of entries in the buffer is greater than a second threshold number.

6. 6. The computer-implemented method of claim 5, wherein the forbidden vertices include a date, a time value, a link to another reference knowledge graph, a web page, or a duplicated or redirected entity.

7. The priority score of an entry in the buffer is: the number of weakly connected components in a knowledge graph that includes the seed graph and the entity associated with the entry; a graph density of the knowledge graph including the seed graph and the entities associated with the entries; and Control parameters 2. The computer-implemented method of claim 1, wherein the priority function is a function of

8. 8. The computer-implemented method of claim 7, further comprising updating the control parameters after adding the entities and links identified in the first set of entries to the seed graph.

9. 9. The computer-implemented method of claim 8, wherein updating the control parameter comprises multiplying the control parameter by a value less than one.

10. For each weakly connected component in the version of the customized knowledge graph, mapping entities in the weakly connected components to vertices in the reference knowledge graph; Identifying up to the first threshold number of entities in the reference knowledge graph by traversing the reference knowledge graph from the mapped vertices in the reference knowledge graph based on the finite state machine; storing the RDF of the identified entity as an entry in the buffer; calculating a priority score for the entry in the buffer using the priority function and the updated control parameters; selecting a second set of entries from the entries in the buffer having the highest priority scores; and generating an updated customized knowledge graph by adding entities and links identified by the second set of entries to the seed graph.

11. 2. The computer-implemented method of claim 1, further comprising, prior to calculating priority scores for the entries in the buffer, performing reservoir sampling on the entries in the buffer to reduce the number of the entries in the buffer.

12. receiving an input utterance; extracting entities from the input utterance; generating knowledge graph embeddings of the extracted entities based on the customized knowledge graph; and classifying the input utterance based on the knowledge graph embeddings of the extracted entities.

13. 13. The computer-implemented method of claim 12, wherein generating the knowledge graph embedding comprises using TransE or an extension of TransE to generate the knowledge graph embedding.

14. 13. The computer-implemented method of claim 12, wherein generating the knowledge graph embedding comprises using TransE and a weighted scoring function to generate the knowledge graph embedding.

15. The step of generating the knowledge graph embeddings includes: using the reference knowledge graph to pre-train the knowledge graph embeddings of the entities extracted from the input utterance; and retraining the knowledge graph embeddings of the entities extracted from the input utterance using the customized knowledge graph and the pre-trained knowledge graph embeddings.

16. classifying the input utterance based on the knowledge graph embeddings of the extracted entities, determining an average knowledge graph embedding of the knowledge graph embeddings of the extracted entities; determining an average GloVe vector of the GloVe vectors of words in the input utterance; and classifying the input utterance based on the average knowledge graph embedding and the average GloVe vector.

17. Classifying the input utterance based on the average knowledge graph embedding and the average GloVe vector includes: generating a concatenated vector by combining the average knowledge graph embedding and the average GloVe vector; performing a principal component analysis on the concatenated vector; and classifying the input utterance based on the principal component analysis.

18. The step of determining the average GloVe vector of the GloVe vectors of the words in the input utterance includes: cleaning the input utterance; generating a GloVe vector for each word in the cleaned input utterance; and aggregating the GloVe vectors of each word in the cleaned input utterance.

19. 1. A non-transitory computer-readable medium storing a plurality of instructions executable by one or more processors, the plurality of instructions, when executed by the one or more processors, causing the one or more processors to perform operations including: receiving a user data set for the application; extracting entities from the user dataset; identifying links between the entities based on the user data set; creating a seed graph representing the entities and the links between them; identifying weakly connected components in the seed graph; For each weakly connected component in the seed graph, mapping entities in the weakly connected components to vertices in a reference knowledge graph; Identifying up to a first threshold number of entities in the reference knowledge graph by traversing the reference knowledge graph from the mapped vertices in the reference knowledge graph based on a finite state machine; storing a Resource Description Framework (RDF) of the identified entity as an entry in a buffer; calculating a priority score for the entry in the buffer; selecting a first set of entries from the entries in the buffer having the highest priority scores; and generating a version of the customized knowledge graph by adding entities and links identified in the first set of entries to the seed graph.

20. 1. A system comprising: one or more processors; a memory coupled to the one or more processors, the memory storing instructions that, when executed by the one or more processors, cause the system to perform operations including: receiving a user data set for the application; extracting entities from the user dataset; identifying links between the entities based on the user data set; creating a seed graph representing the entities and the links between them; identifying weakly connected components in the seed graph; For each weakly connected component in the seed graph, mapping entities in the weakly connected components to vertices in a reference knowledge graph; Identifying up to a first threshold number of entities in the reference knowledge graph by traversing the reference knowledge graph from the mapped vertices in the reference knowledge graph based on a finite state machine; storing a Resource Description Framework (RDF) of the identified entity as an entry in a buffer; calculating a priority score for the entry in the buffer; selecting a first set of entries from the entries in the buffer having the highest priority scores; and generating a version of the customized knowledge graph by adding entities and links identified in the first set of entries to the seed graph.

Images