Techniques for building knowledge graphs in limited knowledge domains

Abstract

The technology disclosed herein generally relates to constructing a customized knowledge graph. In one embodiment, entities and relationships between entities are extracted from a user data set based on certain rules to generate a seed graph. A finite state machine is then used to traverse the large knowledge graph to identify candidate entities and / or relationships to be added to the seed graph. A priority function is used to select entities and / or relationships from the candidate entities and / or relationships. The selected entities and / or relationships are then added to the seed graph to generate a customized knowledge graph.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 765,005, filed on August 16, 2018, entitled “Techniques For Building A Knowledge Graph In Limited Knowledge Domains,” and U.S. Patent Application No. 16 / 542,017, filed on August 15, 2019, entitled “TECHNIQUES FOR BUILDING A KNOWLEDGE GRAPH IN LIMITED KNOWLEDGE DOMAINS,” the entire disclosures of which, including any appendices, are incorporated herein by reference for all purposes. Background Art

[0003] Knowledge graphs organize information in a structured way by explicitly describing the relationships between entities. Knowledge graphs typically represent a knowledge base using a directed graph with labeled edges (representing relationships) between nodes or vertices (representing entities). Knowledge graphs can be used for, for example, search, question answering, conversational interfaces (e.g., chatbots), recommendation systems, etc. There are some large-scale and comprehensive knowledge graphs available. However, natural language understanding (NLU) services for specific applications or environments (such as building conversational interfaces (e.g., chatbots) for different users) typically perform better in specific domains. Creating an accurate knowledge graph for a specific knowledge domain is a challenging task, partly due to the subtlety and ambiguity of natural language and the limited knowledge in a specific domain due to the typically small number of samples in the user data set. Summary of the invention

[0004] The present disclosure generally relates to building customized knowledge graphs, and more particularly to techniques for constructing customized knowledge graphs for specific knowledge domains (e.g., for specific clients or specific robots) based on small user data sets and large-scale reference (e.g., external) knowledge graphs. The customized knowledge graphs can be used, for example, to improve intent classification in chatbots based on knowledge graph embedding techniques. Various invention embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, codes, or instructions executable by one or more processors, and the like.

[0005] In certain embodiments, a computer-implemented method for generating a customized knowledge graph for an application is disclosed. The method may include: receiving a user data set for the application; extracting entities from the user data set; identifying links between entities based on the user data set; and creating a seed graph representing entities and links between entities. The method may also include: identifying weakly connected components in the seed graph, and for each weakly connected component in the seed graph, mapping entities in the weakly connected components to vertices in a reference knowledge graph; starting from the mapped vertices in the reference knowledge graph and traversing the reference knowledge graph based on a finite state machine to identify up to a first threshold number of entities in the reference knowledge graph; and saving the resource description framework (RDF) of the identified entities as entries in a buffer. The method may also include: calculating priority scores for entries in the buffer; selecting a first group of entries with the highest priority scores from the entries in the buffer; and adding entities and links identified by the first group of entries to the seed graph to generate a version of the customized knowledge graph.

[0006] In some embodiments, the user data set may include multiple user utterances. Extracting entities and identifying links between entities may include, for example, performing part-of-speech tagging, named entity recognition, and / or constituency parsing on the user data set. Mapping entities in weakly connected components to vertices in a reference knowledge graph may include mapping entities to vertices in a reference knowledge graph using, for example, Dexter 2.

[0007] In some embodiments, the finite state machine may include a disambiguation state, an entity state, a forbidden state, and an end state. Traversing the reference knowledge graph may include: when the next vertex in the reference knowledge graph is a disambiguation vertex, entering the disambiguation state; when the next vertex in the reference knowledge graph is a forbidden vertex, and if the current state of the finite state machine is not a forbidden state, then entering the forbidden state; when the next vertex in the reference knowledge graph is an entity vertex, entering the entity state, and saving the RDF of the next vertex in the reference knowledge graph as an entry in the buffer; and when the number of entries in the buffer is greater than a second threshold number, entering the end state. Forbidden vertices may include, for example, dates, time values, links to another reference knowledge graph, web pages, or duplicate or redirected entities.

[0008] In some embodiments, the priority scores of the entries in the buffer can be determined using a priority function that is a function of the number of weakly connected components in the knowledge graph including the seed graph and the entities associated with the entries, the graph density of the knowledge graph including the seed graph and the entities associated with the entries, and / or a control parameter. In some embodiments, the method may also include: before calculating the priority scores for the entries in the buffer, performing reservoir sampling on the entries in the buffer to reduce the number of entries in the buffer.

[0009] In some embodiments, the method may further include updating the control parameters after adding the entities and links identified in the first set of entries to the seed graph. Updating the control parameters may include multiplying the control parameters by a value less than 1. In some embodiments, the method may further include, for each weakly connected component in the version of the customized knowledge graph, mapping the entities in the weakly connected components to vertices in the reference knowledge graph; starting from the mapped vertices in the reference knowledge graph and traversing the reference knowledge graph based on a finite state machine to identify up to a first threshold number of entities in the reference knowledge graph; and saving the RDF of the identified entities as entries in a buffer. The method may further include: calculating the priority scores of the entries in the buffer using a priority function and the updated control parameters; selecting a second set of entries with the highest priority scores from the entries in the buffer; and adding the entities and links identified by the second set of entries to the seed graph to generate an updated customized knowledge graph.

[0010] In some embodiments, the method may further include: receiving an input utterance, extracting entities from the input utterance, generating a knowledge graph embedding for the extracted entities based on a customized knowledge graph, and classifying the input utterance based on the knowledge graph embedding of the extracted entities. In some embodiments, generating the knowledge graph embedding may include generating the knowledge graph embedding using TransE or an extension of TransE. In some embodiments, generating the knowledge graph embedding may include generating the knowledge graph embedding using TransE and a weighted scoring function. In some embodiments, generating the knowledge graph embedding may include: pre-training the knowledge graph embedding of the entity extracted from the input utterance using a reference knowledge graph; and re-training the knowledge graph embedding of the entity extracted from the input utterance using a customized knowledge graph and a pre-trained knowledge graph embedding.

[0011] In some embodiments, classifying the input utterance based on the knowledge graph embeddings of the extracted entities may include: determining the average knowledge graph embeddings of the knowledge graph embeddings of the extracted entities; determining the average GloVe vector of the GloVe vectors of the words in the input utterance; and classifying the input utterance based on the average knowledge graph embedding and the average GloVe vector. In some embodiments, classifying the input utterance based on the average knowledge graph embedding and the average GloVe vector may include: combining the average knowledge graph embedding and the average GloVe vector to generate a concatenated vector; performing principal component analysis on the concatenated vector; and classifying the input utterance based on the principal component analysis. In some embodiments, determining the average GloVe vector of the GloVe vectors of the words in the input utterance may include: cleaning the input utterance; generating a GloVe vector for each word in the cleaned input utterance; and averaging the GloVe vectors of each word in the cleaned input utterance.

[0012] According to certain embodiments, a non-transitory computer-readable medium may store a plurality of instructions executable by one or more processors, wherein the plurality of instructions, when executed by the one or more processors, may cause the one or more processors to perform processing, including: receiving a user data set of an application; extracting entities from the user data set; identifying links between entities based on the user data set; and creating a seed graph representing entities and links between entities. The processing may also include: identifying weakly connected components in the seed graph, and for each weakly connected component in the seed graph, mapping entities in the weakly connected components to vertices in a reference knowledge graph; starting from the mapped vertices in the reference knowledge graph and traversing the reference knowledge graph based on a finite state machine to identify up to a first threshold number of entities in the reference knowledge graph; and saving the resource description framework (RDF) of the identified entities as entries in a buffer. The processing may also include: calculating priority scores for entries in the buffer; selecting a first group of entries with the highest priority scores from the entries in the buffer; and adding entities and links identified by the first group of entries to the seed graph to generate a customized version of the knowledge graph.

[0013] According to certain embodiments, a system may include one or more processors and a memory coupled to the one or more processors. The memory may store instructions that, when executed by one or more processors, may cause the system to perform operations including: receiving a user data set of an application; extracting entities from the user data set; identifying links between entities based on the user data set; and creating a seed graph representing entities and links between entities. The operation may also include: identifying weakly connected components in the seed graph, and for each weakly connected component in the seed graph, mapping entities in the weakly connected components to vertices in a reference knowledge graph; starting from the mapped vertices in the reference knowledge graph and traversing the reference knowledge graph based on a finite state machine to identify up to a first threshold number of entities in the reference knowledge graph; and saving the resource description framework (RDF) of the identified entities as entries in a buffer. The operation may also include: calculating priority scores for entries in the buffer; selecting a first group of entries with the highest priority scores from the entries in the buffer; and adding entities and links identified by the first group of entries to the seed graph to generate a customized version of the knowledge graph.

[0014] The foregoing and other features and embodiments will become more fully apparent by reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Illustrative examples are described in detail below with reference to the following figures.

[0016] Figure 1 A distributed system implementing a robotic system for communicating with end users using a messaging application is depicted in accordance with certain embodiments.

[0017] Figure 2 Depicted is an integrated system including a robotic system and a robotic analysis system for monitoring, analyzing, visualizing, and improving performance of the robotic system in accordance with certain embodiments.

[0018] Figure 3 An example of a knowledge graph is illustrated.

[0019] Figures 4A-4D An example of a process of building a knowledge graph is illustrated.

[0020] Figure 5 is a simplified flow chart illustrating an example of a method for building a customized knowledge graph for a specific domain (e.g., for a chatbot) in accordance with certain embodiments.

[0021] Figure 6 Illustrated is an example of a finite state machine (FSM) for controlling traversal of a reference knowledge graph in accordance with certain embodiments.

[0022] Figure 7 An example of a seed graph generated based on a user dataset (eg, user utterances) in accordance with some embodiments is illustrated.

[0023] Figure 8 The diagram shows a method based on some embodiments of the present invention. Figure 7 An example of a customized knowledge graph generated from the seed graph shown in .

[0024] Fig. 9 yes Figure 8 A zoomed-in view of a portion of the custom knowledge graph drawn in .

[0025] Fig.10 Depicted is a simplified diagram of a distributed system for implementing certain embodiments.

[0026] Fig.11 is a simplified block diagram of a cloud-based system environment in which various services may be provided as cloud services according to certain embodiments.

[0027] Fig.12 An example of a computer system that may be used to implement certain embodiments is illustrated. DETAILED DESCRIPTION

[0028] The present disclosure generally relates to building customized knowledge graphs, and more particularly to techniques for constructing customized knowledge graphs for specific knowledge domains using limited knowledge in the domain (e.g., a small user data set) and large-scale reference (e.g., external) knowledge graphs. The customized knowledge graphs are used, for example, to improve intent classification in chatbots based on knowledge graph embedding techniques. Various inventive embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, codes, or instructions executable by one or more processors, and the like.

[0029] Natural language understanding (NLU) services for specific applications or environments, such as conversational interfaces for different users (e.g., chatbots), can perform better using customized knowledge graphs within specific domains rather than large comprehensive knowledge graphs. For example, the noise in a customized knowledge graph may be smaller, so using a small customized knowledge graph rather than a large general knowledge graph may be more efficient and more accurate. However, the available knowledge in a specific domain may be limited, so a customized knowledge graph generated based on limited knowledge may be sparse.

[0030] According to some embodiments, entities for custom applications (e.g., chatbots) can be extracted from a small user data set (including, for example, user utterances, intents, entities, and QnA) based on certain rules. A seed graph can be generated based on the extracted entities and the connections or relationships between the entities. A finite state machine can then be used to traverse a large-scale reference knowledge graph (such as WikiData) to identify candidate entities and / or relationships to be added to the seed graph to expand the seed graph into a custom knowledge graph for a custom application. The traversal can also help identify possible additional relationships between entities and related facts in the user data set from the reference knowledge graph. A scoring function can be used to select entities and / or relationships from the identified entities and / or relationships, and then the selected entities and / or relationships are added to the seed graph to generate a customized knowledge graph. Using certain knowledge graph embedding techniques, the generated customized knowledge graph can be used for customer applications, such as custom chatbots. Ablation studies have shown that the additional semantic information captured by the custom knowledge graph can improve the performance of the custom chatbot. The technology disclosed herein can also be used in other applications, such as question answering or relationship extraction.

[0031] As used herein, a "chatbot," "robot," or "skill" refers to a computer program designed to simulate a conversation with a human user, especially over the Internet. Individual skills can be designed to interact with users and complete specific types of tasks, such as ordering food, making reservations, and changing contact information. Each skill can help users complete tasks by combining text messages and simple UI elements such as a selection list.

[0032] As used herein, the term "intent" may refer to a category of actions or tasks that a user expects a skill to perform for it. The term "entity" may refer to variables that identify key information from user input that enables a skill to complete a task. The term "component" may refer to various functions that a skill can use to respond to a user. The term "component" may refer to common functions such as outputting text or returning information from a backend and executing custom logic. The term "conversation flow" may refer to the definition of a skill-user interaction and may describe how a skill responds and behaves based on user input. The term "channel" may refer to a platform-specific configuration to allow a skill to access a messaging platform or client messaging application. A single skill may have several channels configured for it so that it can run on different services or platforms simultaneously.

[0033] As used herein, an utterance or message may refer to one or more sentences exchanged during a conversation, wherein a conversation may refer to a communication session that may include one or more utterances or messages. A conversation may include one or more phases or states. A conversation flow may be an abstraction of multiple conversations including the same phase or state and the same transition from phase (or state) to phase (or state). Each conversation may be a specific instance of a corresponding conversation flow. The state or phase of a conversation or conversation flow may be associated with the state of a state machine maintained by a robot system for conducting a conversation with other robot systems or people. In some cases, a state may correspond to an end-user's intention or goal. As used herein, an end-user may refer to an end-user of a robot system, such as a person or another entity conducting a conversation with a robot system through a messaging application or platform. For example, an end-user may be a customer or client of an enterprise that owns a robot system. As used herein, a user of an analysis system may refer to an owner, operator, administrator, or developer of a robot system.

[0034] In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of the examples of the present disclosure. However, it will be clear that various examples can be practiced without these specific details. The subsequent description only provides examples and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the subsequent example description will provide a feasible description for implementing the examples to those skilled in the art. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the spirit and scope of the present disclosure as set forth in the appended claims. Each figure and description is not intended to be restrictive. Circuits, systems, networks, processes and other components may be shown as components in block diagram form so as not to obscure the examples in unnecessary details. In other cases, well-known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details to avoid obscuring the examples. The teachings disclosed herein may also be applied to various types of applications, such as mobile applications, non-mobile applications, desktop applications, web applications, enterprise applications, etc. In addition, the teachings of the present disclosure are not limited to specific operating environments (e.g., operating systems, devices, platforms, etc.), but may be applied to multiple different operating environments.

[0035] In addition, it is noted that various examples may be described as processes, which are depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although flow charts may describe operations as sequential processes, many operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. The process may terminate when the operations of the process are completed, but there may be additional steps not included in the figure. The process may correspond to a method, function, procedure, subroutine, subprogram, etc. When the process corresponds to a function, its termination may correspond to the function returning to the calling function or main function.

[0036] The word “example” or “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment or design described herein as “exemplary” or “example” is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0037] The term "machine-readable storage medium" or "computer-readable storage medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying (one or more) instructions and / or data. A machine-readable storage medium or a computer-readable storage medium may include a non-transient medium in which data may be stored and does not include a carrier wave and / or transient electronic signal that is propagated wirelessly or via a wired connection. Examples of non-transient media may include, but are not limited to, a disk or tape, an optical storage medium such as a compact disk (CD) or a digital versatile disk (DVD), a flash memory, a memory, or a memory device. A computer program product may include code and / or machine-executable instructions that may represent any combination of a procedure, function, subroutine, program, routine, subroutine, module, software package, class, or instruction, data structure, or program statement. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0038] Furthermore, the examples may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., a computer program product) for performing the necessary tasks may be stored in a machine-readable medium. (One or more) processors may perform the necessary tasks.

[0039] The systems depicted in some figures may be provided in various configurations. In some examples, the system may be configured as a distributed system, where one or more components of the system are distributed over one or more networks in a cloud computing system.

[0040] Where a component is described as being “configured to” perform certain operations, such configuration may be accomplished, for example, by designing electronic circuits or other hardware to perform the operations, by programming or controlling electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0041] To get an instant response, many users around the world are on instant messaging or chat platforms. Organizations often use these instant messaging or chat platforms to have real-time conversations with customers (or end users). However, it can be very expensive for organizations to hire service personnel to communicate with customers or end users in real time. Therefore, chatbots (also known as robots (bots), skills, chatbots (chatterbots) or conversational (talkbot) robots) have begun to be developed to simulate conversations with end users, especially over the Internet. End users can communicate with robots through messaging applications that end users have already installed and used. Intelligent robots, which are often driven by artificial intelligence (AI), can communicate more intelligently and contextually in real-time conversations, and therefore can make the conversation between the robot and the end user more natural, thereby improving the conversational experience. Instead of the end user learning a fixed set of keywords or commands that the robot knows how to respond to, the intelligent robot may be able to understand the end user's intent based on the user's utterances in natural language and respond accordingly.

[0042] A robotic system may include: one or more user intent classification engines for identifying end-user intent based on end-user utterances; and one or more dialogue engines for intelligently and context-awarely responding to user utterances based on the determined end-user intent. However, building a robotic system (such as a user intent classification engine that can determine end-user intent based on user utterances) is a challenging task, in part due to the subtlety and ambiguity of natural language and the dimensionality of the input space (e.g., possible user utterances) and the size of the output space (the number of intents).

[0043] Robots can usually respond to natural language messages (e.g., questions or comments) using messaging applications of natural language messages. Enterprises can use one or more robot systems to communicate with end users through messaging applications. The messaging application that can be referred to as a channel can be a messaging application preferred by end users that the end user has installed and is familiar with. Therefore, end users can chat with the robot system without downloading and installing new applications. Messaging applications can include, for example, upper-layer (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack or SMS), virtual private assistants (such as Amazon Dot, Echo or Show, Google Home, Apple HomePod, etc.), mobile and web application extensions that extend to local or hybrid / responsive mobile applications or web applications with chat capabilities, or voice-based input (such as, devices or applications with interfaces that interact using Siri, Cortana, Google Voice or other speech inputs).

[0044] In some examples, a robot system can be associated with a uniform resource identifier (URI). The URI can use a string to identify the robot system. The URI can be used as a webhook for one or more messaging applications. The URI can include, for example, a uniform resource locator (URL) or a uniform resource name (URN). The robot system can be designed to receive messages (e.g., hypertext transfer protocol (HTTP) post call messages) from a messaging application system. The HTTP post call message can be directed to the URI from the messaging application system. In some embodiments, the message can be different from the HTTP post call message. For example, the robot system can receive a message from a short message service (SMS). Although the discussion herein may refer to communications received by the robot system as messages, a person of ordinary skill in the art will recognize that the message can be an HTTP post call message, an SMS message, or any other type of communication between two systems.

[0045] End users can interact with the robot system through conversational interactions (sometimes called conversational user interfaces (UIs)), just like interactions between people. In some cases, the interaction can include the end user saying "Hello" to the robot, and the robot responds with "Hi" and asks the end user how to help. In some cases, the interaction can also be a transactional interaction with, for example, a banking robot, such as transferring money from one account to another; an informational interaction with, for example, an HR robot, such as checking the number of days left on a vacation; or an interaction with, for example, a retail robot, such as discussing a return or seeking technical support.

[0046] In some embodiments, the robot system can intelligently respond to end-user interactions without interacting with the administrator or developer of the robot system. For example, the end user can send one or more messages to the robot system to achieve the desired goal. The message may include certain content, such as text, emoticons, audio, images, videos or other methods of conveying messages. In some embodiments, the robot system can convert the content into a standardized form (e.g., a representative state transfer (REST) ​​call for an enterprise service with appropriate parameters) and generate a natural language response. The robot system can also prompt the end user to provide additional input parameters or request other additional information. In some embodiments, the robot system can also initiate communication with the end user instead of passively responding to the end user's speech.

[0047] A conversation with a robot can follow a specific conversation flow that includes multiple states. The flow can define what will happen next based on the input. In some embodiments, a state machine that includes user-defined states (e.g., end-user intent) and actions to be taken in a state or from one state to another can be used to implement a robot system. The conversation can take different paths based on the end-user input, which may affect the decision made by the robot for the flow. For example, in each state, the robot can determine the end-user's intent based on the end-user input in order to determine the next appropriate action to take.

[0048] Intents may include goals that an end user wants to achieve. Intents map end user inputs to actions that the backend system should perform for the end user. Thus, based on the phrases spoken by the end user in natural language, the bot maps the end user utterances to specific use cases or jobs, such as ordering a pizza, getting an account balance, transferring funds, making a purchase, returning an item, and so on. Human conversations are often non-linear in nature. End users may often enter different states during a conversation. For example, if an end user wants to transfer funds from account A to a recipient, the end user may start a conversation with the bot system by, for example, asking the bot to pay the recipient for dinner. The bot may respond with "from which account?" The end user may pick a checking account, but then may realize that he or she is unsure of the balance in that account. Thus, the end user may switch context to ask about balances and recent transactions, and so on. In other words, the end user may trigger changes in flow and state, for example, from transferring funds to checking balances and then to recent transactions. At some point in time, the end user may decide to return to the original intent - paying the recipient. Thus, one task of the bot system is to dynamically determine the end user's intent from natural language utterances.

[0049] The bot can use a natural language processing (NLP) engine and / or a machine learning model (e.g., an intent classifier) ​​to map the end user's utterances to specific intents. For example, a machine learning-based NLP engine can learn to understand and classify natural language conversations from end users and extract necessary information from the conversations so that precise actions can be taken, such as executing a transaction or looking up data from a backend recording system.

[0050] Figure 1 A distributed system 100 is depicted that can be used to implement a robotic system for communicating with end users using a messaging application in accordance with certain embodiments. The system 100 may include a robotic system 120, one or more messaging application systems 115, and one or more end-user devices, such as one or more mobile devices 110. In some examples, the messaging application can be installed on an electronic device (e.g., a desktop computer, a laptop, a mobile device 110, etc.). Although the discussion herein will refer to mobile devices and messaging applications, one of ordinary skill in the art will recognize that any electronic device can be used, and any messaging platform or messaging application can be used, such as Messenger, Instant messaging software, Mobile text and voice messaging communications services, Messenger, Messenger, Messenger, short message service (SMS) or any other messaging application that provides a platform for end users to communicate. In other examples, the browser installed on the mobile device 110 (e.g., Browser, BROWSERS and INTERNET EXPLORER BROWSER) to run messaging applications. In some embodiments, two or more messaging applications can be installed on end-user devices to communicate through two or more messaging platforms (such as, two or more messaging application systems 115).

[0051] The messaging application may be facilitated by a messaging platform such as a messaging application system 115. The mobile device 110 may be connected to the messaging application system 115 via a first network (e.g., the Internet). The messaging application system 115 may be a messaging platform provided by a third party such as Facebook, Tencent, Google, Microsoft, etc. The messaging application system 115 may manage content sent and received via the messaging application on multiple mobile devices or other end-user devices.

[0052] The robot system 120 (e.g., implemented on one or more servers) can also be communicatively connected to the messaging application system 115 to send and receive messages. The communication between the messaging application system 115 and the robot system 120 can be through a second network (e.g., the Internet). The first network and the second network can be the same network, or they can be similar or completely different networks. The messaging application system 115 can use the Internet to route content (e.g., messages or information in messages) from the mobile device 110 to the robot system 120. In some embodiments, the destination of the content (e.g., the identity of the robot system 120) can be included in the content as a nominal addressee. In some embodiments, the robot system 120 can also be configured to communicate with two or more messaging applications 115.

[0053] As described above, content exchanged between end users or between end users and a robot system may include, for example, text, emoticons, audio, media (e.g., pictures, videos, links), or any other method of conveying a message. The robot system 120 receives messages from, for example, Examples of messages received by Messenger may include:

[0054]

[0055] The robot system 120 may receive content from the messaging application system 115 using a connector 130 that acts as an interface between the messaging application system 115 and the robot system 120. In some embodiments, the connector 130 may normalize the content from the messaging application system 115 so that the robot system 120 can analyze the content across different messaging application systems. The content normalization process may include formatting the content from each type of messaging application into a common format for processing. In some embodiments, the robot system 120 may include a common format for each messaging application (such as, Messenger, Instant messaging software, Mobile text and voice messaging communications services, Messenger, Messenger and Messenger, Short Message Service (SMS)). In some embodiments, connector 130 can route content to message-in queue 140. Message-in queue 140 can include a buffer (e.g., a first-in, first-out (FIFO) buffer) that stores content in the order in which it was received. In some embodiments, each connector 130 can be associated with one or more message-in queues.

[0056] When the message processor 150 becomes available, the message input queue 140 can send the content to the message processor 150. In some embodiments, the message processor 150 can extract the content from the message input queue 140. The message processor 150 can parse the message and determine the intent of the parsed message, as described in detail below. In some embodiments, the message processor 150 may include a natural language processor 152 and an intent determination subsystem 154. The natural language processor 152 can parse the message and perform certain semantic analysis, such as identifying the subject, predicate (e.g., action) and / or object. The intent determination subsystem 154 can determine the end-user intent based on the parsed message. As described above, the intent may include the purpose of the message. For example, the purpose of the message may be to order a pizza, order a computer, transfer funds, ask questions about delivery, etc. In some embodiments, parameters (which may be referred to as entities) associated with the intent that more specifically define or clarify the action to be taken may also be extracted from the message by the natural language processor 152 and / or the intent determination subsystem 154.

[0057] After the end-user intent is determined by the message processor 150 based on the content, the determined intent (and parameters associated with the intent) can be sent to the action engine 160. The action engine 160 can be used to determine the action to be performed based on the intent (and parameters associated with the intent) and the current state (or context) of the state machine, as described above. For example, the action engine 160 can send certain outbound content as a response to a message-out queue 170, and / or can send commands to or retrieve information from some enterprise services such as enterprise services 125. The message-out queue 170 can send the outbound content to the connector 130. The connector 130 can then send the outbound content to the messaging application system indicated by the action engine 160, which can be the same as or different from the messaging application system 115. The messaging application system 115 can then forward the outbound content to the messaging application on the mobile device 110.

[0058] The robot system 120 can communicate with one or more enterprise services (e.g., enterprise service 125), one or more storage systems for storing and / or analyzing messages received by the robot system 120, or a content system for providing content to the robot system 120. The enterprise service 125 can communicate with one or more of the connectors 130, the action engine 160, or any combination thereof. The enterprise service 125 can communicate with the connector 130 in a manner similar to the messaging application system 115. The enterprise service 125 can send content to the connector 130 to be associated with one or more end users. The enterprise service 125 can also send content to the connector 130 to cause the robot system 120 to perform an action associated with the end user. The action engine 160 can communicate with the enterprise service 125 to obtain information from the enterprise service 125 and / or instruct the enterprise service 125 to take an action identified by the action engine 160.

[0059] In some embodiments, the robot system 120 may include one or more timers. The timer may cause the action engine 160 to send content to the end user using the connector 130 and the messaging application 115 after a certain amount of time has passed. In some embodiments, the timer may send content to the robot system 120 similar to the end user or enterprise service 125. For example, because the message from the end user is to be analyzed, the timer may send the message to the robot system 120 for analysis.

[0060] In a specific embodiment, the end user can use the mobile device 110 to send a message to the robot system 120 through the messaging application system 115. The message may include a greeting, such as "Hello" or "Hi". The robot system may determine that a new conversation with the end user has begun and start a state machine. In some embodiments, the robot system may identify one or more characteristics of the end user. For example, the robot system may use a profile associated with the end user on the messaging application system to identify the name of the end user. Using one or more characteristics, the robot system may respond to the end user on the messaging application. The response may include a message to the end user that responds to the message received from the end user. For example, the response may include a greeting with the name of the end user, such as "Hi, Tom, What can I do for you?". Depending on the business associated with the robot system, the robot system may progress to achieve the goals of the business. For example, if the robot system is associated with a pizza delivery business, the robot system may send a message to the end user asking if the end user wants to order a pizza. The conversation between the robotic system and the end user can continue from there, back and forth, until the robotic system has completed the conversation or the end user stops responding to the robotic system.

[0061] In some embodiments, the robot system can initiate a conversation with the end user. The conversation initiated by the robot system can be in response to a previous conversation with the end user. For example, the end user may have ordered a pizza in a previous conversation. The robot system can then initiate a conversation when the pizza is ready. In some embodiments, the robot system can determine that the pizza is ready when an indication is received from a business associated with the robot system (e.g., an employee sends a message to the robot system that the pizza is ready). The conversation can include a message sent to the end user indicating that the pizza is ready.

[0062] In some embodiments, the robot system may send a message to the end user on a messaging application different from the messaging application in which the previous message was received. For example, the robot system may determine to use Short Message Service (SMS) instead of Messenger to send messages. In such an embodiment, the robot system can integrate multiple messaging applications.

[0063] In some embodiments, the robot system may determine to start a conversation based on a timer. For example, the robot system may determine to schedule a timer for the end user for one week after ordering a pizza. Expiration of the one-week timer may cause the robot system to start a new conversation with the end user for ordering another pizza. The timer may be configured by the enterprise and implemented by the robot system.

[0064] As described above, in some embodiments, the action engine 160 can send commands to some enterprise services 125 or retrieve information from them. For example, when the robot system 120 (more specifically, the message processor 150) determines the intention to check the balance, the robot system 120 can determine which of several accounts (e.g., checking or savings accounts) needs to check the balance. If the end user enters "What's my balance in my savings account", the robot system 120 can extract "savings" and send a command to the bank server to check the balance, and then send the received balance information to the end user via a message. If the end user initially only says "What's the balance in my account?", the robot system 120 can send a message to the end user, prompting the end user to further specify a specific account, or can retrieve information for all accounts of the end user and send the account information to the end user for the end user to select.

[0065] In some embodiments, the robot system can maintain information between conversations. This information can be used later so that the robot system does not need to ask certain questions every time a new conversation is started between the end user and the robot system. For example, the robot system can store information related to the end user's previous pizza order. In a new conversation, the robot system may send a message to the end user asking if the end user wants the same order as last time.

[0066] In some embodiments, the robot system 120 can store information associated with the end user in a cache. After an outbound message is sent from the connector 130 to the messaging application, the cache can be written to the database to save the information. In other embodiments, the cache can be written to the database at different times (e.g., after a specific event, after each event, after a certain amount of time, or any other metric that determines when to write to the database).

[0067] When a slowdown is identified, the robotic system 120 can allow for scaling of each component. For example, if the robotic system 120 identifies that the number of messages arriving at the connector 130 exceeds a threshold, then one or more additional connectors can be added to the connector 130. In addition, depending on where the slowdown occurs, the number of message input queues, message processors, action engine instances, and message output queues can be increased. In such an embodiment, additional components can be added without having to add other additional components. For example, connectors can be added without having to add additional instances of the action engine. In some embodiments, one or more components or portions of components of the robotic system 120 can run on a virtual machine. By running on a virtual machine, additional virtual machines can be launched as desired.

[0068] As described above, building a robotic system such as a user intent classification engine that can determine an end user's intent based on an end user's utterance is a challenging task, in part due to the subtlety and ambiguity of natural language, as well as the dimensionality of the input space (e.g., possible end-user utterances) and the size of the output space (the number of intents). As a result, it may be necessary to monitor, debug, and modify new robotic systems to improve the performance of the robotic system and the user experience of the robotic system. In many cases, it may be difficult to more specifically identify the root cause of the robotic system's performance being lower than expected and determine how to improve the robotic system without the use of analytical or optimization tools.

[0069] In some cases, a bot owner, developer, or administrator may want to monitor the operational status of the bot and understand how the bot is being used and where end users are abandoning the bot in order to improve the bot. For example, a bot owner, developer, or administrator may want to know which bot conversations are successful and which are not successful in order to identify and diagnose elements of the bot system that are underperforming.

[0070] According to some embodiments, an analysis system can be integrated with a robot system. The analysis system can monitor events that occur during a conversation between an end user and a robot system, aggregate and analyze the collected events, and graphically provide information about the conversation at different levels of summary (such as all conversations, different categories of conversations, and individual conversations) on a graphical user interface. For example, the graphical user interface can display options for filtering or selecting certain types of conversations or individual conversations, and graphically display the selected information, for example by visualizing the path of the conversation. The analysis system can also provide suggestions, options, or other information to improve the robot system.

[0071] Figure 2An integrated system 200 including a robot system (such as the robot system 120) and a robot analysis system for monitoring, analyzing, visualizing, and improving the performance of the robot system is depicted according to some embodiments. As shown, the robot system may include a connector 230 and multiple robot engines, such as a dialog engine 212, an intent modeler 214, an entity resolver 216, and a custom component 218. The robot system may also include a database 240, a management API 250, a user interface 254, and a UI server 252. The robot analysis system may include a collector 255, an enrichment engine 260, a database 270, and a REST server 280. The robot analysis system may also include a user interface 292 and a UI server 290. The collector 255 of the robot analysis system may collect events 205 occurring at the robot system. Feedback 294 from the robot analysis system may be provided to the robot system through the user interface 292 and the user interface 254.

[0072] Connector 230 can act as an interface between the robot system and one or more end users through one or more channels (such as channels 220 and 222). Each channel can be a messaging application, such as a messaging channel (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack or SMS), a virtual personal assistant (such as Amazon Dot, Echo or Show, Google Home, Apple HomePod, etc.), a mobile and web application extension that is extended to a local or hybrid / responsive mobile application or web application with chat capabilities, or a voice-based input (such as a device or application with an interface that interacts using Siri, Cortana, Google Voice or other voice input). In some embodiments, connector 130 can standardize content from different channels so that the robot system can analyze content across different messaging application systems. Content standardization processing can include formatting content from each type of messaging application into a common format for processing. In some embodiments, the robot system may include one or more connectors for each channel.

[0073] The intent modeler 214 can be used to determine the end-user intent associated with the end-user utterance. In some embodiments, the intent modeler 214 for determining the end-user's intent based on one or more messages received by the robot system from the end-user can use a natural language processor to mark parts of speech (verbs, nouns, adjectives), find roots / stems (runs / running / ran->run), and mark entities (Texas->LOCATION). In some embodiments, the intent modeler 214 can standardize messages. For example, "Mary ran to Texas" can become "PERSON run to LOCATION". The intent modeler can also include logic for detecting words with the same meaning within end-user messages. For example, if the training data set includes: "Mary ran to Texas" and "Bob walked to Detroit", both of which are mapped to the same intent, and run / walk appear in the same intent set, then the intent modeler 214 can learn that run = walk for the purpose of intent parsing. In an illustrative example, "Mary ran to Texas" can become "PERSON run to LOCATION", and "Bob walked to Detroit" can become "PERSON walk to LOCATION". In the illustrative example, both sentences can be associated with the same intent because for the purpose of intent parsing, "noun run to noun" is the same as "noun walk to noun". In another example, “I want to order a large cheesepizza” and “I want to order a small pepperonipizza” can both be normalized to “I want to order a pizza of Bots_PizzaSize Bots_Toppings”.

[0074] After normalization, the probability that the occurrence of a word can represent a certain intent can be determined. In some examples, the probabilities can be combined using a basic probability algorithm as if they were independent. For example, if "order" indicates a 20% probability of ordering a pizza, and "pizza" indicates a 10% probability of ordering a pizza, then the total probability would be 1-(1-0.2)(1-0.1)=28%. Some probabilities may be based on the presence of a word, or on the presence of certain linguistic elements, such as negation or personal pronouns.

[0075] Another level of rules can be template rules, which are combinations of words. In some examples, each sentence in the training data set, once standardized, can automatically become a rule. In such examples, the training data set can include a very small number of short sentences. Template rules can return a probability of 1. New rules can be generated from rules via inductive processing. For example, the following sentences may belong to tracking expenses: "How much did I spend last month on gas?" and "How much did I spend in May on food?". These sentences can be used to derive the rule "How much did I spend", because this is a shared part between them. In other examples, the training data set can include the phrase "How much did I spend" to achieve the same result.

[0076] The example described above allows the definition of intents to not include duplicates, such as variations of named entities (e.g., "Send money to Sue" and "Send money to Bob"). However, similar sentences where one or two words are different can be used for training. Similar sentences can allow the model to learn which words are likely to have the same meaning for intent parsing, and which words are likely to be common misspellings.

[0077] If a particular word or group of words (e.g., verbs) is important to an intent, the probability can be manipulated by having more examples using such words (and their synonyms) and fewer examples of such words for other intents.

[0078] Examples can also be provided to prevent the model from making wrong assertions. For example, a specific subphrase or a word that only appears for a certain intent may lead to a wrong assertion. Similarly, the model can be prevented from synthesizing broad rules using similar sentences belonging to different intents used for training.

[0079] The entity resolver 216 can identify entities (such as objects) associated with the end-user intent. For example, in addition to the end-user intent (such as "order pizza") identified by the intent modeler 214, the entity resolver 216 can also resolve entities associated with the intent, such as pizza types, pizza toppings, etc.

[0080] The dialogue engine 212 can be used to handle the dialogue between the end user and the robot system. For example, the dialogue engine 212 can respond to the end user utterance based on the end user intent identified by the intent modeler 214 and the entity associated with the end user intent identified by the entity resolver 216. In some embodiments, the dialogue engine 212 can use a state machine to handle the dialogue with the end user, which includes user-defined states (such as end user intent) and actions to be taken in a state or from one state to another state.

[0081] Custom components 218 may include custom modules for a particular robot system. For example, a financial robot may include custom components that may be used, for example, to check balances, transfer funds, or pay bills.

[0082] The database 240 can be used to store data of the robot system, such as data for the classification model, conversation logs, etc. An administrator or developer of the robot system can use the management API 250 to manage the robot system, such as retraining the classification model, editing intent, or otherwise modifying the robot system. An administrator or developer can use the user interface 254 and the UI server 252 to manage the robot system.

[0083] When the robot system is running, various events may be generated. Events may be generated based on one or more instructions included in the robot system. For example, an event may be generated when the robot system enters a specific state, where the specific state is defined by an administrator or developer of the robot system. As events are generated, the robot analysis system may collect, store, and analyze the events. When an event is captured, additional information associated with the event may also be collected, where the additional information may indicate the current context in which the event was generated.

[0084] For example, a dialog event may be generated by the dialog engine 212. A dialog event may include a message (referred to as msg_received) received by the robot system from an end-user device. Msg_received may include one or more of the following parameters or variables: the content of the message, the time when the robot system receives the message, the language of the received message, device characteristics (e.g., version or name), operating system characteristics (e.g., version or name), geographic location characteristics (e.g., Internet Protocol address, latitude, longitude, etc.), identification information (e.g., user ID, session ID, robot system ID, tenant ID, etc.), timestamp (e.g., timestamp when the device is created, the device is sent, the collector is derived), channel, etc.

[0085] Dialog events may also include messages sent by the robot system to the end-user device (referred to as msg_sent). Msg_sent may include one or more of the following: the content of the message (e.g., the text or HTML of the message), the time when the robot system sent the message, the language of the message, the creator of the message (e.g., the robot system or the end-user device), device characteristics, operating system characteristics, browser characteristics (e.g., version or name), application characteristics (e.g., version or name), geographic location characteristics (e.g., Internet Protocol address, latitude, longitude, etc.), identification information (e.g., user ID, session ID, robot system ID, tenant ID, etc.), channel (e.g., Facebook or Webhook), etc.

[0086] The dialogue engine 212 may also generate dialogue state execution events. As described above, the dialogue engine 212 may use a state machine to determine the flow of dialogue with the end user. The state machine may include a set of states and transition rules between states. The dialogue engine 212 may execute the state machine for each end user dialogue, and may generate dialogue state execution events for each state that the dialogue engine 212 gradually passes through to process the end user's utterance. The attributes of the dialogue state execution event may include, for example, state name, component name, next action, entity match, intent match, variable, user query statement, response statement, execution time, communication language, device characteristics, operating system characteristics, geographic location characteristics, identification information, timestamp, channel, etc. The state name may be the name of the state currently being executed or an "error state". The component name may be the name of the robot component executed for the current state. The next action may be the next action to be executed. The entity match may be the entity resolved in the current message. The intent match may be the intent resolved with a score value. The variable may be the variable value of the current state. The query statement may be a message sent by the end user. The response statement may be a message sent to the end user. The execution time may be the timestamp of the completed state execution. The communication language may be the language of the message being talked about. Device and / or operating system characteristics may be associated with an end user interacting with a robotic system. Browser and / or application characteristics may be associated with an end user interacting with a robotic system. Geographic location characteristics may be the location of an end user interacting with a robotic system.

[0087] Intent resolution events may occur as a result of execution of the intent modeler 214. The intent modeler 214 may use a trained or otherwise defined classification model to identify an end-user intent from a set of intents based on an end-user utterance. The results of the intent classification may be captured as intent resolution event attributes, which may include, for example, a final intent classification result (such as an identified intent) and a confidence score associated with each corresponding intent in the intent set.

[0088] The entity resolver 216 can generate entity resolver events. An entity is an object associated with an end-user intent. When creating a robot system, entity definition rules can be determined. For example, in addition to resolving end-user intents such as "order pizza", the robot system can also use the entity resolver 216 to resolve associated entities such as pizza type, pizza ingredients, etc. Entity resolver events can be captured when the entity is resolved. Examples of attributes associated with entity resolver events can include entity name, applied rules, search terms, resolved state, query statements, entity types, execution time, communication language, device characteristics, operating system characteristics, browser characteristics, application characteristics, geographic location characteristics, identification information, timestamps, channels, etc. The entity name can be the name of the entity currently being resolved. The rules applied can be, for example, before, after, or summary. The search terms can be from, to, destination, origin, etc. The resolved state can be the state of the conversation for entity resolution. The query statement can be a message containing entity values. The entity type can be system or derived. The execution time can be the timestamp of entity resolution. The communication language may be the language of the conversational messages. The device and / or operating system characteristics may be associated with the end user interacting with the robotic system. The browser and / or application characteristics may be associated with the end user interacting with the robotic system. The geographic location characteristics may be the location of the end user interacting with the robotic system.

[0089] Custom components may also generate events, such as predefined events or custom events. Predefined events may be characteristics captured when executing a custom component. Examples of properties of predefined events may include: component name, event name, payload, execution time, communication language, device characteristics, operating system characteristics, browser characteristics, application characteristics, geographic location characteristics, identification information, timestamp, channel, etc. The component name may be the name of the custom component currently being executed. The event name may be invoked, invocation_failed, replied, replied_failed, etc. In the case of failure, the payload may be the reason for the failure, a stack trace, etc. The execution time may be a timestamp indicating when the event occurred. The communication language may be the language of the message being chatted. The device and / or operating system characteristics may be associated with the end user interacting with the robot system. The browser and / or application characteristics may be associated with the end user interacting with the robot system. The geographic location characteristics may be the location of the end user interacting with the robot system.

[0090] The custom component may also emit custom events during the execution of the custom component. Examples of properties of the custom event may include component name, event name, custom payload, execution time, communication language, device characteristics, operating system characteristics, browser characteristics, application characteristics, geographic location characteristics, identification information, timestamp, channel, etc. The component name may be the name of the custom component currently being executed. The event name may be a user-defined event name (e.g., Balance_Retrieved). The payload may be, for example, {"amount":"USD 100","account":"checking"}. The execution time may be a timestamp indicating when the event occurred. The communication language may be the language of the message being spoken. The device and / or operating system characteristics may be associated with the end user interacting with the robot system. The browser and / or application characteristics may be associated with the end user interacting with the robot system. The geographic location characteristics may be the location of the end user interacting with the robot system.

[0091] Error events and timeout events can also be generated by the robot system during execution. Error events can be generated when an error occurs. Timeout events can be generated when the end-user conversation is inactive for a period of time, which can be configured at the channel.

[0092] The robot analysis system can collect events and additional information when the robot system conducts a dialogue with an end user and generates corresponding events. For example, the collector 255 can collect events and additional information and send the collected information to a queue. In some embodiments, the collector 255 can be configurable and can be programmed to collect the above-mentioned different events and / or event attributes as needed. For example, the collector 255 can be configured to capture dialogue state attributes, intent resolution attributes, entity resolution attributes, and error and timeout attributes. In some embodiments, the collector 255 can also be configured to collect information about events 295 generated by systems other than the robot system.

[0093] The expansion engine 260 can perform validation and expansion on the collected events and other information and write them to the database 270. For example, based on the collected IP address, the expansion engine 260 can determine the location of the end user associated with the IP address. As another example, the expansion engine 260 can extract certain features from the collected information, such as determining the web browser or channel used by the end user. The REST server 280 can analyze the expanded events and other information and generate various reports based on certain summary metrics 272. The reports can be displayed to the owner, administrator, or developer of the robot system on the user interface 292 through the UI server 290. The owner, administrator, or developer of the robot system can provide feedback 294 to the robot system to improve the robot system.

[0094] As described above, building a robot system that can determine the intention of an end user based on user speech, such as a user intent classification engine (e.g., message processor 150 or intent modeler 214), is a challenging task, in part due to the subtlety and ambiguity of natural language, the dimensionality of the input space (e.g., possible user speech) and the size of the output space (the number of intents). In some robot systems, training speech and associated intent can be used to train the robot to recognize intent based on user speech. For example, in a robot system for a shoe store that may focus on capturing the user's "buy shoes" intent, the high-level intent of "buy shoes" can be fed into the user intent classification engine, and training phrases such as "I want to buy some shoes", "I am looking for a pair of shoes", "I want shoes" and the like can also be provided to train the user intent classification engine. In some embodiments, the user intent classification engine can try to expand the example phrases and use the example phrases to match the user speech. For example, a user intent classification engine may be able to match the phrase "I want to purchases boots" with the intent "purchase shoes" because a natural language processor (e.g., natural language processor 152) may understand that the semantic distance between boots, shoes, or even slippers is small and they all refer to the same category of things. Such a user intent classification engine may be suitable for some chatbot interactions, but may not be able to cope with more complex tasks.

[0095] Instead of using intent matching as described above, some robotic systems can combine lower-level analysis results (syntactic analysis, entities, key phrases) with a domain-specific knowledge graph to identify user intent. A knowledge graph (or ontology) can connect intents, phrases, nouns, and adjectives to concepts in a domain. For example, a robot can use a natural language processor to analyze input from a user and use the analysis results to browse the knowledge graph and find user intent at a semantic level. The user intent can then be used to determine the most appropriate next response to the user.

[0096] The knowledge graph acquires information and integrates it into an ontology, which includes a set of concepts and categories in a subject scope or domain, and displays its characteristics and the relationships between them. In some cases, the knowledge graph can also apply a reasoner to derive new knowledge. The knowledge graph can capture facts about places, people, things, processes, applications, and the relationships between them, and use such information to provide, for example, more relevant information to users performing searches or queries. The knowledge graph typically uses a directed graph to represent the knowledge base, which has labeled edges (representing relationships) between nodes or vertices (representing entities). For example, a knowledge graph can be based on a set of triples in the form of (subject, predicate, object), such as (Obama, born in Hawaii), (Brazilian football team, has position, goalkeeper), etc.

[0097] Figure 3 An example of a knowledge graph 300 is illustrated. The knowledge graph 300 may include a plurality of vertices 310 representing, for example, places (such as Paris), people (such as Bob, Bill, and Leonardo da Vinci), things (such as the Eiffel Tower and Mona Lisa), concepts (e.g., time, age, dimensions, and categories of things, such as cities, people, and places), processes, applications, and the like. The vertices 310 are connected by labeled edges 320 that describe the relationship between the two vertices in each pair of connected vertices. The edges 320 may be unidirectional or bidirectional. By traversing the knowledge graph 300 along the path formed by the edges 320, one vertex 310 may be retrieved from another vertex 310.

[0098] Figures 4A-4D An example of a process of building a knowledge graph based on a knowledge base such as a book or Wikipedia is illustrated. Figure 4A In , a string 410 (e.g., "Leonardo da Vinci") may be obtained from a knowledge base. Figure 4B In , entity 420 (e.g., "Leonardo da Vinci") can be identified and recognized from string 410. Figure 4C In , entities 432 related to entity 420 can be identified from the knowledge base and added to graph 430. Figure 4D In the embodiment of the present invention, the relationship between entities 432 can be determined from the knowledge base and added to the graph as edges 442 or connections between entities 432 to form a knowledge graph 440 including multiple entities 432 and edges 442. Entities 432 and edges 442 can be represented by triples of (subject, predicate, object), (entity, attribute, value) or other forms. The labels on the edges 442 can indicate the relationship between the entities 432.

[0099] The knowledge graph 440 can be used as a seed graph to build a larger knowledge graph. For example, related entities of the entity 432 in the knowledge graph 440 and the relationship between the related entities and the entity 432 in the knowledge graph 440 can be identified or determined from a knowledge base or a reference knowledge base (e.g., an external knowledge base) and added to the knowledge graph 440 to further expand the knowledge graph 440. The expansion of the knowledge graph 440 can continue until, for example, the knowledge in a given field is exhausted, which can generate a knowledge graph for the given field.

[0100] Applications of knowledge graphs in natural language understanding include QnA, such as answering questions by embedding relevant parts of the knowledge graph, or generating relevant query subgraphs using reward functions learned by deep convolutional neural networks. Knowledge graphs can also be used, for example, by search engines to provide more relevant information in response to user searches, or by chatbots to perform intent classification as described above.

[0101] Chatbots may perform better in specific domains where there is sufficient data. For a given bot, the user-provided dataset may only include, for example, about 50-100 utterances. This limited dataset may not be sufficient to build a knowledge graph that can be used by the bot to work correctly. Reference knowledge graphs or knowledge bases, such as WikiData, Freebase, DBpedia, etc., can be used to extend the knowledge in a specific knowledge domain and generate customized knowledge graphs.

[0102] In some embodiments, the knowledge graph may be expanded by, for example, randomly walking a reference knowledge graph or other knowledge base, and selecting information from the reference knowledge graph or knowledge base based on, for example, the frequency with which nodes of the reference knowledge graph or knowledge base have been visited. However, for applications such as chatbots, the quality of the knowledge graph generated by random walks may be insufficient. For example, the knowledge graph generated by random walks may be noisy.

[0103] According to certain embodiments, a method for building a customized knowledge graph for a given domain (e.g., a specific robot) using a customer dataset (including, for example, user utterances, intents, entities, and questions and answers (QnA)) and a reference (e.g., external) knowledge base (such as WikiData) is disclosed. The customized knowledge graph can be domain-specific and can be generated in two main stages. First, entities, such as user-provided utterances, can be extracted from the customer dataset and used as vertices or nodes of an initial knowledge graph, which can be used as an expanded seed graph. These extracted entities and the links between them can be used to form a seed graph to initialize the customized knowledge graph. In the second stage, the seed graph can be extended by traversing the reference knowledge graph using a finite state machine, where the entities in the seed graph can be mapped to the reference knowledge graph as the starting vertex for traversing the reference knowledge graph and selecting candidate entities to be added to the customized knowledge graph. The candidate entities selected from the reference knowledge graph can then be filtered based on their priority scores added to the customized knowledge graph.

[0104] Figure 5 is a simplified flow chart 500 illustrating an example of a method for building a customized knowledge graph (such as for a chatbot) according to some embodiments. Figure 5 The processes presented in and described below are intended to be illustrative and non-limiting. Figure 5 Various operations that occur in a particular order or sequence are depicted, but this is not intended to be limiting. In certain alternative embodiments, operations may be performed in some different order, or some operations may be performed in parallel. In addition, embodiments of the method may be implemented in hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments for performing associated tasks may be stored in a computer-readable medium such as a storage medium. One or more processors may perform the associated tasks.

[0105] At 502, a computing system may receive a user dataset. The user dataset may include, for example, user utterances, user-defined entities and intents, user-provided questions and answers, etc. As described above, in some embodiments, the user dataset may include, for example, about 50 to about 100 utterances, each of which may or may not have an associated intent tag.

[0106] At 504, entities can be extracted from a user dataset such as user utterances using techniques such as part-of-speech (PoS) tagging (see, e.g., Kristina Toutanova et al., “Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network,” Proceedings of HLT-NAACL 2003, pp. 252-259), named entity recognition (NER) (see, e.g., Jenny Rose Finkel et al., “Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling,” Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL 2005), pp. 563-370), composition parsing using, e.g., CoreNLP (see, e.g., Manning et al., “The Stanford CoreNLP Natural Language Processing Toolkit,” Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: Systems Processing, 1996), and the like. Demonstrations, 2014, pp. 55-60), etc. In some embodiments, entity linking techniques such as Dexter 2 techniques can be used to detect entities from user datasets (see, for example, Ceccarelli et al., "Dexter: an open source framework for entity linking", Proceedings of the sixth international workshop on Exploiting semantic annotations in information retrieval, ACM 2013, pp. 17-20).

[0107] At 506, the relationship between the identified entities can be identified based on the user data set, for example, using the techniques described above for extracting entities. The relationship can be used to link or connect the identified entities to generate a seed graph, which includes the identified entities (as nodes or vertices in the seed graph) and the relationship between the identified entities (as edges between vertices). The edge can be a directed edge, including the direction of the edge between entities. The edge can be unidirectional or bidirectional. The seed graph can include a small number of vertices and can be a sparse graph that may not include all relationships between entities. The seed graph can include one or more weakly connected components (WCCs), where WCCs generally refer to the largest subgraph of a directed graph, so that any node can be reached from any other node by traversing the edge along an undirected path (i.e., ignoring the direction of the edge). The nodes in the weakly connected components can all have an out-degree or in-degree of at least 1. In contrast, a strongly connected component can refer to a subgraph in which there is a directed path from each node to each other node in the subgraph.

[0108] At 508-516, a simulated annealing optimization method (see, e.g., Nourani & Andresen, "A comparison of simulated annealing cooling strategies", J. Phys. A: Math. Gen. 51. 1998, 8373-8385) can be used to iteratively expand the seed graph by gradually adding entities and relationships to the seed graph. In each iteration, a temperature value can be set or adjusted at 508. As described in detail below, the temperature can be used to determine the priority score of the candidate entity to be added to the knowledge graph.

[0109] At 510, for each weakly connected component in the seed graph, each entity (i.e., vertex) in the weakly connected component can be mapped to the reference knowledge graph and used as a starting vertex for traversing the reference knowledge graph and identifying up to M candidate entities using a finite state machine as described in more detail below. The resource description framework (RDF) of the identified candidate entities can be saved as entries in a priority queue.

[0110] At 512, the entries in the priority queue may be scored using a scoring function that is a function of temperature. In some embodiments, the total number of candidate entities identified and added to the priority queue at 510 may be greater than some threshold number N. Techniques such as reservoir sampling techniques (see, e.g., J. Vitter, "Random sampling with a servoir", ACM Trans. on Mathematical Software 11(1):37-57, March 1985) may be performed to reduce the number of entries in the priority queue to N, thereby reducing the time used to calculate the priority scores of the entries, which may be computationally more expensive than a knowledge graph traversal. Priority scores may be calculated for each of the N candidate entities in the N entries and the corresponding paths to the candidate entities, as described in more detail below.

[0111] At 514 , K entries may be selected from up to N entries in the priority queue having the highest scores, and corresponding entities and relationships identified by the K entries may be added to the seed graph to extend the seed graph.

[0112] At 516, if it is determined that the temperature is above the threshold, then the operations at 508-514 can be performed again in the next iteration, during which the temperature (and therefore the scoring function) can be changed, and the entities in the weakly connected components may be different because the seed graph has been modified using the top K entries in the priority queue. When the temperature is below the threshold, the optimization of the customized knowledge graph can end, and the customized knowledge graph can be used for a user application associated with the user data set, such as a specific robot or search engine.

[0113] Table 1 shows an example of pseudo code for implementing a method for generating a customized knowledge graph, such as the method described above. Figure 5The method described. The pseudocode in Table 1 shows an example of a finite state machine for traversing a reference knowledge graph. In Table 1, the first line of the pseudocode involves identifying entities and links between entities based on a user data set (e.g., user utterances). Line 2 of the pseudocode involves creating a seed graph. Line 3 is used to set the initial conditions for the simulated annealing process to expand the seed graph to generate a customized knowledge graph, including initializing the temperature and the priority queue. Lines 4-23 involve operations performed at each temperature of the simulated annealing process, wherein lines 5-15 describe the use of a finite state machine to traverse the reference knowledge graph to identify candidate entities and save the RDF triples of the candidate entities as entries in a priority queue, which will be described in more detail below. Lines 16-17 involve reducing the number of entries in the priority queue when the number of entries in the priority queue is greater than a threshold number N to perform priority score calculations. Lines 18-22 involve calculating the priority scores of the entries in the priority queue and using the top K entries with the highest priority scores to expand the seed graph. Update the temperature at line 23.

[0114] Table 1 Knowledge graph generation

[0115]

[0116] The above is relative to Figure 3 The methods described in Table 1 may generally include: (1) extracting entities from the utterances provided by the user and constructing a seed graph based on the extracted entities and links between the entities, and (2) extending the seed graph by repeatedly traversing the reference knowledge graph using a finite state machine, each of which will be described in more detail below.

[0117] From a user dataset such as a set of raw sentences (e.g., utterances), entities can be extracted and used as knowledge graph vertices, and existing relationships between entities can be determined and used as edges between vertices in the knowledge graph. For example, the above description of Figure 5504 describes part-of-speech tagging, named entity recognition, and composition parsing using CoreNLP technology to generate a list of candidate entities. PoS tagging technology includes tagging words with their appropriate part of speech (such as noun, verb, adjective, adverb, pronoun, etc.). Named entity recognition (also known as entity identification, entity chunking, or entity extraction) includes locating and classifying named entities in unstructured text into predefined categories, such as names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc. Composition analysis includes extracting a composition-based parse tree from a sentence to represent the syntactic structure of the sentence according to a phrase structure grammar. For example, a composition-based parse tree for the sentence "John hit the ball" may start with a sentence (S) and end in each leaf node (John, hit, the, ball), where "John" may be a noun (N) node, "hit" may be a verb (V) node (a transitive verb in this example), "theball" may be a noun phrase (NP) as a branch node, "the" may be a determiner (D or DT) node, and "ball" may be a noun (N) node.

[0118] In some embodiments, when generating a candidate entity list, the following pre-processing rules may be applied: (a) Check whether the NER is not empty and whether the word is a noun phrase (e.g.,

[0119] (a) checking for matches (extracted entities) where a proper noun (NNP, which names a specific person, place, thing, or idea, e.g., "John") is dominated by a NP (i.e., the NP is the parent of the NNP in the semantic parse tree); (b) checking for matches (extracted entities) where a common noun (NN, which is a general noun that names an unspecific person, place, thing, or idea, e.g., "desk") is dominated by a NP (i.e., the NP is the parent of the NN in the semantic parse tree); (c) checking for matches (extracted entities) where a common noun (NN, which is a general noun that names an unspecific person, place, thing, or idea, e.g., "desk") is dominated by a NP (i.e., the NP is the parent of the NN in the semantic parse tree); (d) stripping personal pronouns (PRP, e.g., "I", "he", or "she") and determiners (DT) from the above; (e) dividing the NER phrase into chunks; and (f) removing numbers and dates from the NER phrase. For example, applying the above rule to the sentence "How much did I spend from my checking account on ServiceA?" may produce a list of candidate entities including ['checking account', 'ServiceA']. In another example, applying the above rule to the sentence "I want to go to an outdoor cafe on Pioneer Day" may produce a list of candidate entities including ['Pioneer Day', 'outdoor cafe', 'outdoor cafe on Pioneer Day'].

[0120] In some embodiments, the Dexter 2 technique discussed above can also be used to generate a list of candidate entities by detecting text snippets that reference entities in parallel. The two lists of candidate entities can then be combined to generate a list of candidate entities. The combined list of candidate entities can then be used to generate a seed graph for knowledge graph generation.

[0121] In some embodiments, Dexter 2 technology can also be used to link candidate entities to their corresponding entries in a reference knowledge graph such as WikiData, where Dexter 2 technology can be used to link candidate entities to, for example, Wikipedia entries, which can then be mapped to WikiData entries. For example, entities extracted from the example sentence "How much did I spend from my checking account on ServiceA?" (and their corresponding WikiData IDs) may include "ServiceA" (Q780442) and "checking account" (Q695430). In another example, entities extracted from the example sentence "Michael Jordan played basketball at an outdoor cafe on Pioneer Day" (and their WikiData IDs) may include "café" (Q30022), "Pioneer Day" (Q254295), "basketball" (Q5372), "Michael Jordan" (Q41421), and "outdoor" (Q3312129). In some embodiments, the Mediawiki Webservice API can be used to search for entities that do not match the above method.

[0122] Table 2 illustrates an example of a method (such as the above method) for extracting entities from a user dataset (e.g., utterances) and building a seed graph. In operation 1 shown in Table 2, for example, entities are extracted from the original sentence of the user utterance using part-of-speech tagging, named entity recognition, or composition parsing using the CoreNLP technology described above.

[0123] Table 2 Methods for extracting entities and constructing seed graphs

[0124]

[0125] In operation 2, as described above, a set of pre-processing rules may be applied to generate a candidate entity list. The pre-processing rules may include, for example, (a) checking whether the named entity recognition is not empty and whether the word is part of a noun phrase; (b) checking matches (extracted entities) where proper nouns are dominated by NPs (i.e., NPs are parents of NNPs in the semantic parse tree); (c) checking matches (extracted entities) where common nouns are dominated by NPs (i.e., NPs are parents of NNs in the semantic parse tree); (d) stripping pronouns (PRPs) and determiners (DTs) from the above; (e) dividing NER phrases into chunks; and (f) removing numbers and dates from NER phrases. The final list after the above operations is a set of results for each individual rule.

[0126] In operation 3, a graph search query may be determined based on the above preprocessing rules, according to the results of the discourse parsing in operation 1 and a candidate entity list generated by an entity linking technique (such as the above Dexter 2.0 framework). As used herein, entity linking may generally refer to matching an entity with a correct knowledge base entry when there is ambiguity. As an example, from the sentence "On July 20, 1969, the Apollo 11 astronauts Neil Armstrong, Michael Collins, and Edwin 'Buzz' Aldrin Jr. realized President Kennedy's dream", the following graph search queries can be determined: "Apollo 11" (space mission), "astronauts", "Neil Armstrong" (astronaut), "Michael Collins" (astronaut), "Edwin 'Buzz' Aldrin" (astronaut) and John F. Kennedy. In another example, from the sentence "Michael Jordan played for the Chicago Bulls", the following graph search queries can be determined: "Michael Jordan" (basketball player) and "Chicago Bulls". Dexter The Dexter 2 technique can also be used to link graph lookup queries to their corresponding entries in a reference knowledge base such as a reference knowledge graph (e.g., WikiData). For example, a graph lookup query can be linked to a Wikipedia entry using the Dexter 2 technique, and the Wikipedia entry can then be mapped to a WikiData entry.

[0127] In operation 4, for each entity that is not matched with a reference knowledge base entry using, for example, Dexter 2.0, a call can be made to the Mediawiki Webservice API to perform entity linking and / or obtain information from the reference knowledge base. In some embodiments, in operation 5, the returned WikiData entity ID can be cached, for example, using Redis to reduce future API calls.

[0128] In operation 6, for all entities in each utterance, all possible combinations (N choose 2 combinations) can be checked. If there is a link between two entities in the utterance, then the link can be added to the seed graph. At the end of operation 6, a seed graph G = (V, E) can be generated, where the vertex V includes the set of identified entities and the edge E includes the link between any two entities in the entity. The link check between two entities can also be performed on the entities in the entire user data set.

[0129] As described above, the user data set may be small, such as about 50-100 utterances. Therefore, the seed graph may include only a small number of entities, such as about 50-100 entities. In addition, the links between entities in the seed graph may be sparse. The seed graph may need to be expanded to include additional information that is useful for a given domain (such as, a specific chatbot). According to certain embodiments, a finite state machine can be used to traverse a reference knowledge graph such as WikiData to identify additional information (e.g., entities and links) that can be added to the seed graph to generate a customized knowledge graph. In some embodiments, a priority queue and a priority function can be used to determine entities and / or links to be added to the knowledge graph. In some embodiments, a simulated annealing process can be used to iteratively optimize the customized knowledge graph. The pseudocode shown in Table 3 describes an example method.

[0130] Table 3. Extended seed graph

[0131]

[0132]

[0133] In operation 1 of the method shown in the pseudo code in Table 3, the simulated annealing process may be started by setting an initial temperature and initializing a priority queue (such as creating an empty priority queue).

[0134] In operation 2(a), each weakly connected component w in the seed graph G g Each entity in can be mapped to a reference knowledge graph, such as WikiData, and a finite state machine can be used to control the reference knowledge graph from weakly connected components w g , to identify candidate entities and store them as entries in a priority queue buffer (e.g., a cache). For example, if the traversed vertex is a disambiguation vertex, the state machine can proceed to the disambiguation state; if the traversed vertex is a forbidden vertex, the state machine can proceed to the forbidden state; and if the traversed vertex is an entity vertex, the state machine can proceed to the entity state; and if for each weakly connected component w g, the entity state has been visited M times, then the state machine can proceed to the "end" state. As used herein, a disambiguation vertex refers to a knowledge graph entry or knowledge base entry that distinguishes multiple entries that share the same or similar tags. As used herein, a prohibited vertex refers to a vertex in a knowledge graph (e.g., WikiData) that increases the sparsity of a custom knowledge graph or does not provide any meaningful information to the custom knowledge graph and is therefore not suitable for addition to a custom knowledge graph. Prohibited vertices can include, for example, dates and / or times (e.g., durations, time points, etc.), links to other reference knowledge base identifiers (IMDb, Freebase, etc.), Wikimedia projects, articles and list pages, pages outside the main Wikimedia knowledge tree, and duplicate and redirected entries.

[0135] Figure 6 An example of a finite state machine (FSM) 600 for controlling the traversal of a reference knowledge graph in each iteration of a simulated annealing process according to some embodiments is illustrated. At a start state 610 of the FSM 600, the traversal may start from a vertex in the reference knowledge graph, where the vertex may correspond to a weakly connected component w in the seed graph G. g The entity w in .

[0136] If the next vertex connected to the starting vertex is a disambiguation vertex, then the FSM 600 can move to the disambiguation state 640. For example, whenever a disambiguation vertex in WikiData is reached, the disambiguation state 640 is reached. If the next vertex is not a disambiguation vertex, but a forbidden vertex, then the FSM 600 can move to the forbidden state 630. Any entry in the reference knowledge graph (e.g., WikiData) that is not a disambiguation vertex or a forbidden vertex is an entity w and can cause the FSM 600 to move to the entity state 620, where the resource description framework (RDF) of entity w can be added to a priority queue buffer (such as a cache). The RDF can be, for example, a triple in the form of subject-predicate-object. The subject can indicate a resource, the object can indicate a quality or aspect of the resource, and the predicate can indicate a relationship between the subject and the object.

[0137] All possible transitions between entity, disambiguation, and forbidden states can be allowed, except for the transition from forbidden state 630 back to forbidden state 630, because any path that passes through forbidden vertices is not desired to be added to the knowledge graph. Through this process, close neighbors of entities in the user dataset (and seed graph) can be identified, disambiguation vertices can be used to expand the graph search, and forbidden vertices can be ignored. When for each weakly connected component w gWhen the entity state 620 has been visited M times, or when there are no more directly or indirectly linked vertices to traverse, the end state 650 may be reached. The traversal of the reference knowledge graph in the iteration may stop at the end state 650.

[0138] In one example, if the starting vertex has a link to the vertex "Obama" in the reference knowledge graph, then the FSM 600 can move from the starting state 610 to the entity state 620, and the RDF for the vertex "Obama" can be added to the priority queue buffer (e.g., pq_cache). If the vertex "Obama" has 2 vertices linked to it: "America" ​​(e.g., linked by the relationship "president of") and "20January2009" (e.g., linked by the relationship "start time"), then when traversing the vertex "20January 2009", the FSM 600 can enter the forbidden state 630 because the date and time vertex is a forbidden vertex. The next vertex traversed is the vertex "America", which is a valid entity. Therefore, FSM 600 may reenter entity state 620 and add the RDF triple for "America" ​​to the priority queue buffer pq_cache. This process may be performed with links to "America" ​​and / or weakly connected components w in the seed graph G. g Repeat for the vertices of other entities in the entity state 620 until the entity state 620 has been visited M times or there are no other directly or indirectly linked vertices to traverse.

[0139] In operation 2(b) of Table 3, a priority score may be calculated for each entry in the priority queue buffer using a scoring function that is a function of temperature. In some embodiments, if the total number of entities that have been visited during an iteration exceeds a threshold N, then the above steps regarding Figure 5 512 describes a reservoir sampling technique to reduce the number of entities in the priority queue buffer to N, thereby reducing the calculation of the priority scores. This may be done for time optimization purposes, since calculating the priority scores may be computationally more expensive than traversing the knowledge graph using a finite state machine.

[0140] For example, in operation 2(a) of the iteration, after identifying candidate entities and saving the corresponding RDF triples to the priority queue buffer, a priority score can be calculated for each entity e in the priority queue and the corresponding path taken to reach entity e according to the priority function:

[0141]

[0142] Where WCC(G’) is the weak connected component function for identifying the weak connected components in a graph; t is the current time step; T(t) is the temperature at time t; k, η, and Z are real constants; and G’ is a new graph formed by the union of the edge sets between graph G and (w, e). The new graph G’ can be expressed as:

[0143] G′ = G ∪ (w,e) (2)

[0144] ρ(G) corresponds to the graph density and can be determined according to the following equation:

[0145]

[0146] Where G is a graph with edges E and vertices V, |E| is the number of edges in graph G, and |V| is the number of vertices in graph G.

[0147] In operation 2(c) of Table 3, an entry with the highest priority score in the priority queue can be selected and added to the seed graph G. In operation 2(d), the temperature of the simulated annealing process can be updated to initiate a new iteration of the process in operation 2. The temperature at time step t can be determined, for example, by the following equation:

[0148] T(t) = αT(t - 1), α ∈ [0.85, 0.96] (4)

[0149] The cooling schedule can be selected based on the impact of the priority function shown in Equation (1). The priority function shown in Equation (1) can preferentially add new vertices at high temperatures to increase the number of entities in the weak connected components. As the system cools, the priority function can place more weight on the graph density to add more edges between the vertices that already exist in the graph. The two parts of the priority function can be normalized by some coefficients (e.g., Z and η) so that they have similar orders of magnitude.

[0150] Figure 7 An example of a seed graph 700 generated based on a user dataset (e.g., user utterances) according to certain embodiments as described above with respect to, for example, Table 2 is illustrated. In this example, the seed graph 700 depicts the relationship between a web application and its parent company. The seed graph 700 can be based on RDF triples, such as <SoftwareA (Software A), part of (is part of), CompanyA (Company A)> (converted from WikiData ID to WikiData label). As shown, the seed graph 700 can be a sparse graph with few entities 710 and few links 720 between the entities 710. The links 720 are labeled with the relationships between the entities 710. Figure 7The sparsity of seed map 700 is shown for illustration purposes only, and the specific contents of seed map 700 are not intended to be easily readable.

[0151] Figure 8 An example of a customized knowledge graph 800 generated based on the seed graph 700 and the techniques described above with respect to, for example, Table 3 is illustrated. Fig. 9 is an enlarged view of portion 810 in knowledge graph 800 . Figure 8 and Fig. 9 For illustrative purposes only, and the specific contents of the knowledge graph 800 are not intended to be easy to read. Compared to the seed graph 700, there may be many more entities 820 and links 830 between entities 820 in the knowledge graph 800. The knowledge graph 800 includes various information about web applications and surrounding concepts. Some examples include:<Internet Message Access Protocol,part of,email> ,<email filtering,instance of,anti-spam> ,<RunKeeper,instance of,mobile app> ,<RunKeeper,operating system,Android> and<CompanyA,country,United States of America> The knowledge graph 800 may also include some information that may not be immediately relevant to the domain of the dataset, such as, for example,<alternativerock,subclass of,rock music> and<calendar,instance of,chronology> Such information, potentially from other domains, can make the graph more generalizable when unseen data is provided.

[0152] Some examples of RDF triples that form the knowledge graph 800 (in the format of "subject|predicate|object") may include "CompanyA Street View|developer|CompanyA", "email filtering|instance of|anti-spam", "BugMeNot|instance of|service on internet", "CompanyB|country|United States of America", "CompanyC.com|software engine|ProductC", "RunKeeper|operating system|Android", "RunKeeper|instance of|mobile app", "ServiceA|instance of|online dating service", "Internet Message Access Protocol|part of|email", "software release life cycle|instance of|software engineering", "CompanyD|owned by|CompanyE", "CompanyF|parent organization|CompanyG", "G Suite|industry|email", "ProductA|publisher|CompanyH", "ProductA|developer|CompanyH", and so on.

[0153] Table 4 illustrates the statistical results of an example of a knowledge graph before and after expansion using a reference knowledge graph. Statistics include, for example, the number of vertices, the number of edges, the graph density, and the number of weakly connected components in the dataset. As shown in Table 4, for a dataset with a large number of starting vertices, the above technique may not add many new vertices, but may add many more edges. For SNIPS, both the number of vertices and the number of edges increase, but the graph density decreases because SNIPS is not a limited knowledge domain dataset. However, for smaller seed graphs, a large number of new vertices may be added. This may be because, for larger datasets, the entities identified for the seed graph may be located in close proximity on the reference knowledge graph. In addition, when there are many starting vertices, when traversing the reference knowledge graph, many new vertices may not be discovered, but new relationships between these entities may be discovered. On the other hand, for smaller datasets and therefore smaller seed graphs, the technique is more likely to discover more new vertices.

[0154] Table 4 Statistics

[0155]

[0156] Knowledge graphs can be used in many applications related to natural language understanding. For example, knowledge graph embedding (KGE) techniques can embed components of a knowledge graph, including entities and relationships, into a continuous vector space to simplify manipulation while preserving the inherent structure of the knowledge graph. Entity and relationship embeddings can then be used, for example, for knowledge graph completion, relationship extraction, entity classification, entity resolution, question answering, etc.

[0157] In general, KGE technology represents entities and relationships in a continuous vector space, and then uses a distance or similarity-based scoring function to measure the rationality of each fact and other information (e.g., entity type, relationship path, text description, or logical rule). Then, entity and relationship embeddings can be obtained by maximizing the overall rationality of the observed facts. Knowledge graph embeddings can contain more semantic meanings than word vectors and can be used to enhance feature representations based on word vectors. KGE can capture semantic representations of entities beyond typical word vectors by, for example, capturing the semantic meaning of multi-word symbols (e.g., phrases) and improving the predictive power of datasets with out-of-vocabulary (OOV) words (such as words that do not have word vectors, especially in small datasets with a large number of unique and important words).

[0158] As mentioned above, a chatbot is a computer program or application that can conduct a conversation via auditory or textual methods by simulating how a person would behave as a conversation partner. Chatbots can be used in conversation systems for a variety of practical purposes, including customer service or information acquisition. Some simpler chatbots can scan for keywords in the input and then extract the response with the best matching keyword or the most similar wording pattern from a database. Some chatbots can use more sophisticated natural language processing techniques to simulate people more realistically.

[0159] According to certain embodiments, knowledge graph embeddings may be used for intent classification, where the task may include, for example, determining by a chatbot what action a user wishes to perform given an input natural language utterance. The chatbot may use both word vectors and knowledge graph embeddings extracted from a custom knowledge graph, such as that generated using the above method, to improve the performance of user intent classification.

[0160] According to certain embodiments, distance transformation techniques such as the TransE technique (see, e.g., Bordes et al., “Translating Embeddings for Modeling Multi-relational Data,” Advances in Neural Information Processing Systems 26. Curran Associates, 2013, pp. 2787-2795) and extensions thereof can be used to pre-train embeddings on WikiData (see, e.g., Vrandecic & "Wikidata: A Free Collaborative Knowledgebase", Communications of the ACM 57:10, 2014, pp. 78-85), because transformation-based modeling techniques are well suited for intent recognition applications. TransE is an energy-based model for learning low-dimensional embeddings of entities, where relations are represented as transformations in an embedding space: if (h, r, t) holds (e.g., is correct RDF), then the embedding of the tail entity t is close to the sum of the embedding of the head entity h and the vectors that depend on the relation r (t ≈ h + r); otherwise, the embedding of the tail entity t is far from the sum of the embedding of the head entity h and the vectors that depend on the relation r.

[0161] KGE techniques (e.g., distance transform or similarity based KGE techniques) typically involve three operations: (1) representing entities and relations, (2) defining scoring functions, and (3) learning entity and relation representations. The first operation specifies the form in which entities and relations are represented in a continuous vector space. Entities are typically represented as vectors, such as fixed points in a vector space. Relations are typically viewed as operations in a vector space, which can be represented as vectors, matrices, tensors, multivariate Gaussian distributions, or Gaussian mixtures. In the second operation, the scoring function f is transformed into r (h, t) is applied to each fact (h, r, t) to measure its plausibility. For example, in the TransE technique, the scoring function can be defined as the negative distance between h+r and t:

[0162] f r (h, t) = -∥h + r - t∥ 1 / 2 . (5)

[0163] Observed facts in a knowledge graph are usually scored higher than unobserved facts. The third operation involves an optimization process that maximizes the total plausibility of observed facts to learn entity and relation representations (e.g., embeddings).

[0164] In some embodiments, for all entities and correspondences in the graph G, the embeddings may be first initialized using pre-trained embeddings, and then the embeddings may be retrained using techniques such as the TransE technique and different scoring functions. For example, the scoring function used for the retraining process may be changed from equation (5) to:

[0165] f r (h, t) = -θ r ∥h + r - t∥ 1 / 2 , (6)

[0166] where θ r is the weight of the entities and relations belonging to the generated graph. This weighted scoring function is inspired by TransM (see, e.g., M. Fan et al., "Transition-based knowledge graph embedding with relational mapping properties", in Proc. 28th Pacific Asia Conf. Language Inf. Comput., 2014, pp. 328-337), and gives greater weight to entities and relations in the graph G. The maximum margin hinge loss function can be used following the open world training paradigm of TransE. The embedding can be updated according to the following gradient:

[0167]

[0168] where ∥h∥2=1 and ∥t∥2=1. Equation (7) can include a maximum margin hinge loss function, where the hyperparameter γ is the margin (>0). Additional regularization terms are included in Equation (7) to prevent the entity and relationship embeddings of the graph G from becoming too sparse, where λ is the regularization strength (>0). T and T' refer to the training triples and corrupted triples in each mini-batch, respectively. This is modeled as a transfer learning process, where the retraining of the graph embeddings can fine-tune the vectors of the generated knowledge graph by giving greater weight to the entities and relations present therein. This process can allow reducing noise in the embeddings and compensating for the weakness of TransE in modeling non-one-to-one relationships by prioritizing the unlikely composition of such non-one-to-one relationships in the generated graph.

[0169] Table 5 shows an example of a method of using the knowledge graph embedding described above for, e.g., intent classification.

[0170] Table 5 Intent classification using knowledge graph embedding

[0171]

[0172] In the example shown in Table 5, the embedding can be pre-trained on, for example, Wikidata at operation 1. At operation 2, for all entities and corresponding relationships in the knowledge graph G, the embedding can be initialized with the pre-trained embedding obtained at operation 1, and then the embedding can be retrained using techniques such as TransE techniques and different weighted scoring functions. At operation 3, for a given discourse, all entities in the discourse can be extracted using the entity extraction and linking process described above with respect to Table 2. Then, all entity embeddings (calculated as above) of the entities in the sentence are averaged. Sentences can also be cleaned in parallel (e.g., pruning, removing vocabulary, removing numbers and unknown characters, etc.) to reduce sparsity, and the GloVe vectors of the remaining words in the sentence (see, for example, Pennington et al., "Glove: Global vectors for word representation", Proceedings of the Empirical Methods in Natural Language Processing (EMNLP), 2014) can be averaged. Then, the average GloVe vector and the average knowledge graph embedding of the sentence are concatenated, and then principal component analysis (PCA) can be performed on the concatenated vector, as shown in, for example, Goikoetxea et al., "Single or multiple? combining word representations independently learned from text and wordnet", Proceedings of AAAI, 2016. At operation 4, the concatenated vector after PCA can be passed to a classifier for intent classification.

[0173] Table 6 illustrates the training time and query time using the knowledge graph embedding technique and the average GloVe vector technique described above with respect to Table 5. The KGE training time is for the first run without caching. Subsequent runs will be faster.

[0174] Table 6 Training and query time

[0175]

[0176] Ablation studies were also performed to determine which aspects of the overall processing may contribute to the improvement in accuracy and F1 scores. Five variants of the technique have been experimented with. In variant Var 1, no hyperparameter optimization is performed (i.e., no grid search is performed), and hyperparameters are randomly selected within a certain range. In variant Var 2, the knowledge graph is initialized with vertices and only edges (relationships) between entities found in the input utterance. In variant Var 3, a one-hop random walk traversal of a reference knowledge graph with a maximum depth of 5 is used instead of a finite state machine as described in the present disclosure. In variant Var 4, the feature vectors are generated by averaging the GloVe vectors for each word in the input utterance. In variant Var 5, the knowledge graph embedding is not retrained as described above with respect to Table 5, but instead uses the TransE embedding pre-trained for WikiData. The F1 score and accuracy results for these five variants are shown in Table 7 and compared with the results obtained without applying any of these variants.

[0177] Table 7 Accuracy data in ablation study

[0178]

[0179] The results of the ablation study show that while hyperparameter optimization may bring small improvements in accuracy (as shown in Var 1 vs. the full algorithm in Table 7), it is not significant compared to the contributions of other techniques disclosed in this article. The performance degradation of random walk traversal (variant Var 3) and average GloVe vector (variant Var 4) indicates that knowledge graph embeddings can capture more information relevant to the classification process. In particular, the drop in F1 score is more obvious than the drop in accuracy. This may indicate that there may be more semantic information about the discourse within the feature vector, which may produce fewer false positives and false negatives. Both the method of initializing the knowledge graph and the method of traversing the reference knowledge graph can significantly affect the results. Retraining knowledge graph embeddings can also improve performance, indicating that retraining embeddings to facilitate the composition of the generated knowledge graph can help model entities and relationships.

[0180] The techniques described herein may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores), hardware, or a combination thereof. The software may be stored in a memory (e.g., on a non-transitory computer-readable storage medium such as a memory device). The techniques described herein may be implemented in a variety of different environments including cloud environments (e.g., various types of clouds including private, public, and hybrid cloud environments), on-premises environments, hybrid environments, and the like.

[0181] Fig.10A simplified diagram of a distributed system 1000 for implementing an embodiment is depicted. In the illustrated embodiment, the distributed system 1000 includes one or more client computing devices 1002, 1004, 1006, and 1008 coupled to a server 1012 via one or more communication networks 1010. The client computing devices 1002, 1004, 1006, and 1008 may be configured to execute one or more applications.

[0182] In various embodiments, server 1012 may be adapted to run or provide one or more services or software applications that enable automatic generation of relationships or mappings between user interface components and data objects in an application using requirements information for the application. For example, in some embodiments, server 1012 may receive requirements information for an application, where the requirements information may include a description of desired functionality for the application and may also include an image of a desired GUI for the application. Server 1012 may then use the requirements information to generate a mapping between user interface components of the application and one or more data objects.

[0183] In some embodiments, the server 1012 may also provide other services or software applications that may be provided in non-virtualized and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as under a software as a service (SaaS) model, to users of client computing devices 1002, 1004, 1006, and / or 1008. Users operating client computing devices 1002, 1004, 1006, and / or 1008 may in turn utilize one or more client applications to interact with the server 1012 to utilize the services provided by these components.

[0184] exist Fig.10 In the depicted configuration, the server 1012 may include one or more components 1018, 1020, and 1022 that implement the functions performed by the server 1012. These components may include software components, hardware components, or a combination thereof that may be executed by one or more processors. It should be appreciated that a variety of different system configurations are possible that may differ from the distributed system 1000. Thus, Fig.10 The embodiment shown in is one example of a distributed system for implementing an embodiment system and is not intended to be limiting.

[0185] According to the teachings of the present disclosure, a user can use client computing devices 1002, 1004, 1006, and / or 1008 to provide required information to server 1012 and receive information (e.g., mapping information) from server 1012. The client device can provide an interface that enables a user of the client device to interact with the client device. The client device can also output information to the user via the interface. Although Fig.10Only four client computing devices are depicted, but any number of client computing devices may be supported.

[0186] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors and other sensing devices, etc. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Apple or UNIX-like operating systems, Linux or Linux-like operating systems such as Google Chrome TMTM OS), including various mobile operating systems (e.g., Microsoft Windows Windows Android TM , Palm Portable handheld devices may include cellular phones, smart phones (e.g., ), tablets (e.g. ), personal digital assistants (PDAs), etc. Wearable devices can include Google Head-mounted displays and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices (e.g., with or without Gesture Input Devices Microsoft Game consoles, Sony System, by Various game systems provided by and others), etc. The client device may be able to execute a variety of different applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications), and may use various communication protocols.

[0187] The communication network(s) 1010 may be any type of network familiar to those skilled in the art, and may support data communications using any of a variety of available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internetwork Packet Exchange), The communication network(s) 1010 may be, by way of example only, a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., in any of the Institute of Electrical and Electronics Engineers (IEEE) 1002.11 protocol suites, and / or any other wireless protocol operating under the network) and / or any combination of these and / or other networks.

[0188] The server 1012 may be composed of one or more general-purpose computers, dedicated server computers (including, for example, PC (personal computer) servers, The server 1012 may be composed of a server, a mid-range server, a mainframe computer, a rack-mounted server, etc.), a server farm, a server cluster, or any other suitable arrangement and / or combination. The server 1012 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization, such as one or more flexible pools of logical storage devices that can be virtualized to maintain a virtual storage device of the server. In various embodiments, the server 1012 may be suitable for running one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0189] The computing system in the server 1012 can run one or more operating systems, including any of the operating systems discussed above and any commercially available server operating system. The server 1012 can also run any of a variety of additional server applications and / or middle-tier applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, Servers, database servers, etc. Exemplary database servers include but are not limited to (International Business Machines) and other commercially available database servers.

[0190] In some implementations, server 1012 may include one or more applications to analyze and integrate data feeds and / or event updates received from users of client computing devices 1002, 1004, 1006, and 1008. As examples, data feeds and / or event updates may include, but are not limited to, data received from one or more third-party information sources and continuous data streams. feed, Updates or real-time updates, which may include real-time events related to sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. Server 1012 may also include one or more applications that display data feeds and / or real-time events via one or more display devices of client computing devices 1002, 1004, 1006, and 1008.

[0191] The distributed system 1000 may also include one or more data repositories 1014, 1016. In some embodiments, these data repositories may be used to store data and other information. For example, one or more data repositories 1014, 1016 may be used to store information, such as requirement information for one or more applications, mapping information, and other information used or generated by the server 1012. The data repositories 1014, 1016 may reside in a variety of locations. For example, a data repository used by a server 1012 may be local to the server 1012, or may be remote from the server 1012 and communicate with the server 1012 via a network-based or dedicated connection. The data repositories 1014, 1016 may be of different types. In some embodiments, the data repository used by the server 1012 may be a database, such as a relational database, such as a database provided by Databases provided by companies and other suppliers. One or more of these databases may be adapted to respond to SQL formatted commands to implement storage, updating, and retrieval of data in the database. The data repositories used by the application may be of different types, such as, for example, key-value repositories, object repositories, or general repositories supported by a file system.

[0192] In certain embodiments, features described in this disclosure may be provided as a service via a cloud environment. Fig.11 is a simplified block diagram of a cloud-based system environment in which various services may be provided as cloud services according to certain embodiments. Fig.11 In the illustrated embodiment, cloud infrastructure system 1102 may provide one or more cloud services that may be requested by a user using one or more client devices 1104, 1106, and 1108. Cloud infrastructure system 1102 may include one or more computers and / or servers, which may include those described above with respect to server 812. The computers in cloud infrastructure system 1102 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0193] The network(s) 1110 can facilitate communication and data exchange between the client devices 1104, 1106, and 1108 and the cloud infrastructure system 1102. The network(s) 1110 can include one or more networks. The networks can be of the same or different types. The network(s) 1110 can support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0194] Fig.11 The embodiment depicted in FIG. 1 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be appreciated that in some other embodiments, the cloud infrastructure system 1102 may have more Fig.11 The components shown may be more or fewer components, may be combined with two or more components, or may have a different configuration or arrangement of components. Fig.11 Three client computing devices are depicted, but any number of client computing devices may be supported in alternative embodiments.

[0195] The term "cloud service" is generally used to refer to services that are made available to users on demand by a service provider's system (e.g., cloud infrastructure system 1102) and via a communications network such as the Internet. Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own internal servers and systems. The cloud service provider's system is managed by the cloud service provider. Customers can therefore take advantage of cloud services provided by a cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, a cloud service provider's system can host applications, and users can order and use the applications on demand via the Internet without having to purchase infrastructure resources for executing the applications. Cloud services are designed to provide easy, scalable access to applications, resources, and services. Several providers offer cloud services. For example, Cloud Infrastructure Systems, Inc. of Redwood Shores, California, offers cloud services. The company provides several cloud services such as middleware services, database services, Java cloud services, etc.

[0196] In some embodiments, the cloud infrastructure system 1102 may provide one or more cloud services using different models such as a software as a service (SaaS) model, a platform as a service (PaaS) model, an infrastructure as a service (IaaS) model, and other models including hybrid service models. The cloud infrastructure system 1102 may include a set of applications, middleware, databases, and other resources that enable the provision of various cloud services.

[0197] The SaaS model enables applications or software to be delivered to customers as a service over a communication network such as the Internet, without the customer having to purchase hardware or software for the underlying application. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by the cloud infrastructure system 1102. Examples of SaaS services provided by the company include, but are not limited to, services for human resource / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytical services, social applications, and other services. In some embodiments, the ability to automatically generate work applications from application demand information can be provided as a service under the SaaS model.

[0198] The IaaS model is often used to provide infrastructure resources (eg, servers, storage devices, hardware, and networking resources) to customers as cloud services to provide elastic computing and storage capabilities. The company provides a variety of IaaS services.

[0199] The PaaS model is typically used to provide platform and environment resources as a service, which enables customers to develop, run, and manage applications and services without having to purchase, build, or maintain such resources. Examples of PaaS services provided by the Company include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), Data Management Cloud Service, various application development solution services, and other services.

[0200] Cloud services are typically provided on an on-demand self-service, subscription-based, elastically scalable, reliable, highly available, and secure basis. For example, a customer may order one or more services provided by the cloud infrastructure system 1102 via a subscription order. The cloud infrastructure system 1102 then performs processing to provide the services requested in the customer's subscription order. For example, a user may provide the cloud infrastructure system with application requirement information and request the cloud infrastructure system to automatically generate a working application based on the requirement information, including automatically generating a mapping between the user interface components and data objects of the application as described above. The cloud infrastructure system 1102 may be configured to provide one or more cloud services.

[0201] The cloud infrastructure system 1102 can provide cloud services via different deployment models. In a public cloud model, the cloud infrastructure system 1102 can be owned by a third-party cloud service provider, and the cloud services are provided to any general public customer, where the customer can be an individual or an enterprise. In certain other embodiments, under a private cloud model, the cloud infrastructure system 1102 can be operated within an organization (e.g., within an enterprise organization) and provide services to customers within the organization. For example, customers can be various departments of an enterprise, such as a human resources department, a payroll department, etc., or even individuals within an enterprise. In certain other embodiments, under a community cloud model, the cloud infrastructure system 1102 and the services provided can be shared by several organizations in a related community. Various other models, such as a hybrid of the models mentioned above, can also be used.

[0202] Client devices 1104, 1106, and 1108 may be of different types (such as Fig.11 1104, 1106, and 1108) and may be capable of operating one or more client applications. A user may use a client device to interact with the cloud infrastructure system 1102, such as requesting services provided by the cloud infrastructure system 1102. For example, a user may use a client device to provide application requirement information to the cloud infrastructure system 1102 and request the cloud infrastructure system to automatically generate a working application based on the requirement information, including automatically generating a mapping between user interface components and data objects of the application as described in the present disclosure.

[0203] In some embodiments, the processing performed by cloud infrastructure system 1102 to provide the requested service may involve big data analysis. Such analysis may involve using, analyzing, and manipulating large data sets to detect and visualize various trends, behaviors, relationships, etc. within the data. The analysis may be performed by one or more processors, possibly processing data in parallel, performing simulations using the data, etc. The data used for the analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0204] As in Fig.11 As depicted in the embodiment of , cloud infrastructure system 1102 may include infrastructure resources 1130 for facilitating the provision of various cloud services provided by cloud infrastructure system 1102. Infrastructure resources 1130 may include, for example, processing resources, storage or memory resources, networking resources, and the like.

[0205] In some embodiments, in order to facilitate efficient provisioning of these resources to support the various cloud services provided by the cloud infrastructure system 1102 for different customers, the resources can be bundled into a collection of resources or resource modules (also referred to as a "pod"). Each resource module or cluster can include a pre-integrated and optimized combination of one or more types of resources. In some embodiments, different clusters can be pre-provisioned for different types of cloud services. For example, a first set of clusters can be provisioned for database services, and a second set of clusters can be provisioned for Java services, where the second set of clusters can include a different combination of resources than the pods in the first set of clusters. For some services, the resources allocated to provision the service can be shared between services.

[0206] The cloud infrastructure system 1102 itself may internally use services 1132 that are shared by different components of the cloud infrastructure system 1102 and facilitate the service provisioning of the cloud infrastructure system 1102. These internally shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services for enabling cloud support, email services, notification services, file transfer services, and the like.

[0207] The cloud infrastructure system 1102 may include multiple subsystems. These subsystems may be implemented in software or hardware or a combination thereof. Fig.11 As shown, the subsystem may include a user interface subsystem 1112 that enables users or customers of the cloud infrastructure system 1102 to interact with the cloud infrastructure system 1102. The user interface subsystem 1112 may include a variety of different interfaces, such as a web interface 1114, an online store interface 1116 (where cloud services provided by the cloud infrastructure system 1102 are advertised and available for purchase by consumers), and other interfaces 1118. For example, a customer may use a client device to request (service request 1134) one or more services provided by the cloud infrastructure system 1102 by placing a subscription order using one or more of the interfaces 1114, 1116, and 1118. These interfaces may include, for example, an online store user interface 1116, where a customer accesses and browses the online store, identifies cloud services provided by the cloud infrastructure system 1102, and places a subscription order for one or more of the services provided that the customer wishes to subscribe to. Other interfaces may include the web interface 1114 and other interfaces 1118.

[0208] In some embodiments, a service request or subscription request may include information identifying a requesting user or customer and one or more services that the customer desires to subscribe to. For example, a customer may place a subscription order for an automated application generation service provided by cloud infrastructure system 1102.

[0209] In some embodiments, such as Fig.11 In the illustrated embodiment, the cloud infrastructure system 1102 may include an order management subsystem (OMS) 1120 configured to process new subscription orders. As part of this process, the OMS 1120 may be configured to: create an account for the customer (if not already created); receive billing and / or accounting information from the customer, which will be used to bill the customer for providing the requested service to the customer; verify the customer information; after verification, book the order for the customer; and orchestrate various workflows to prepare the order for provision.

[0210] Once correctly verified, OMS1120 can call order provisioning subsystem (OPS) 1124, which is configured to provision resources for the order, including processing resources, memory resources, and networking resources. Provisioning can include allocating resources to the order, and configuring resources to facilitate the services requested by the customer order. The manner in which resources are provisioned for the order and the type of resources provided can depend on the type of cloud service that the customer has ordered. For example, according to a workflow, OPS1124 can be configured to determine the specific cloud service requested and identify multiple clusters that may have been pre-configured for the specific cloud service. The number of clusters allocated for the order can depend on the size / quantity / level / scope of the requested service. For example, the number of clusters to be allocated can be determined based on the number of users supported by the service, the duration of the service being requested, etc. The allocated clusters can then be customized for the specific requesting customer to provide the requested service.

[0211] Cloud infrastructure system 1102 can send a response or notification 1144 to the requesting client to indicate when the requested service is ready. In some cases, information (e.g., a link) can be sent to the client so that the client can begin to use and utilize the benefits of the requested service.

[0212] The cloud infrastructure system 1102 can provide services to multiple customers. For each customer, the cloud infrastructure system 1102 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the order, and providing the requested services to the customer. The cloud infrastructure system 1102 can also collect usage statistics about the customer's use of the subscribed services. For example, statistics can be collected for the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and system downtime. This usage information can be used to bill the customer. Billing can be performed, for example, on a monthly cycle.

[0213] The cloud infrastructure system 1102 may provide services to multiple customers in parallel. The cloud infrastructure system 1102 may store information of these customers, including potentially proprietary information. In certain embodiments, the cloud infrastructure system 1102 includes an identity management subsystem (IMS) 1128 that is configured to manage the information of the customers and provide separation of the managed information so that information associated with one customer cannot be accessed by another customer or commingled with another customer's information. The IMS 1128 may be configured to provide various security-related services, such as identity services, such as information access management, authentication and authorization services, services for managing customer identities and roles and related capabilities, and the like.

[0214] Fig.12 An exemplary computer system 1200 that can be used to implement certain embodiments is illustrated. For example, in some embodiments, the computer system 1200 can be used to implement any of the various servers and computer systems described above, such as Figure 1 The model generation system 122 depicted in FIG. Fig.12 As shown, computer system 1200 includes various subsystems, including a processing subsystem 1204 that communicates with multiple other subsystems via a bus subsystem 1202. These other subsystems may include a processing acceleration unit 1206, an I / O subsystem 1208, a storage subsystem 1218, and a communication subsystem 1224. Storage subsystem 1218 may include non-transitory computer-readable storage media, including storage media 1222 and system memory 1210.

[0215] The bus subsystem 1202 provides a mechanism for enabling the various components and subsystems of the computer system 1200 to communicate with each other as desired. Although the bus subsystem 1202 is schematically shown as a single bus, alternative embodiments of the bus subsystem can utilize multiple buses. The bus subsystem 1202 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, etc. For example, such architectures can include an industry standard architecture (ISA) bus, a microchannel architecture (MCA) bus, an enhanced ISA (EISA) bus, a video electronics standard association (VESA) local bus, and a peripheral component interconnect (PCI) bus, which can be implemented as a mezzanine bus manufactured according to the IEEE P1386.1 standard, etc.

[0216] The processing subsystem 1204 controls the operation of the computer system 1200 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processor may include a single-core or multi-core processor. The processing resources of the computer system 1200 may be organized into one or more processing units 1232, 1234, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some embodiments, the processing subsystem 1204 may include one or more dedicated coprocessors, such as a graphics processor, a digital signal processor (DSP), etc. In some embodiments, some or all of the processing subsystem 1204 may be implemented using custom circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs).

[0217] In some embodiments, the processing unit in the processing subsystem 1204 can execute instructions stored in the system memory 1210 or on the computer-readable storage medium 1222. In various embodiments, the processing unit can execute various program or code instructions, and can maintain multiple concurrently executed programs or processes. At any given time, some or all of the program code to be executed may reside in the system memory 1210 and / or on the computer-readable storage medium 1222, including possibly on one or more storage devices. Through appropriate programming, the processing subsystem 1204 can provide the various functions described above. In the case where the computer system 1200 is executing one or more virtual machines, one or more processing units can be assigned to each virtual machine.

[0218] In certain embodiments, a processing acceleration unit 1206 may optionally be provided for performing customized processing or for offloading some of the processing performed by the processing subsystem 1204 , thereby speeding up the overall processing performed by the computer system 1200 .

[0219] The I / O subsystem 1208 may include devices and mechanisms for inputting information to the computer system 1200 and / or for outputting information from or via the computer system 1200. In general, use of the term "input device" is intended to include all possible types of devices and mechanisms for inputting information to the computer system 1200. User interface input devices may include, for example, keyboards, pointing devices such as a mouse or trackball, a touch pad or touch screen incorporated into a display, scroll wheels, click wheels, dials, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include software such as Microsoft Windows that enables a user to control and interact with input devices. Motion sensing and / or gesture recognition devices, Microsoft 360 game controllers, devices that provide an interface for receiving input using gestures and spoken commands. User interface input devices may also include eye gesture recognition devices, such as detecting eye activity from a user (e.g., a "wink" when taking a picture and / or making a menu selection) and translating eye gestures to input devices (e.g., Google ) Blink detector. In addition, the user interface input device may include a user to enable the user to communicate with the voice recognition system (e.g., Navigator) interactive voice recognition sensing device.

[0220] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or indicator rods, game pads and graphic tablets, and audio / video devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers 3D scanners, 3D printers, laser rangefinders, and eye gaze tracking devices. In addition, user interface input devices may include, for example, medical imaging input devices such as computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0221] In general, use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from the computer system 1200 to a user or other computers. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices, etc. The display subsystem may be a cathode ray tube (CRT), a flat panel device such as one utilizing a liquid crystal display (LCD) or plasma display, a projection device, a touch screen, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0222] The storage subsystem 1218 provides a repository or data repository for storing information and data used by the computer system 1200. The storage subsystem 1218 provides an example of a tangible, non-transitory computer-readable storage medium for storing basic programming and data constructs that provide the functionality of certain embodiments. The storage subsystem 1218 can store software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 1204, provides the functionality described above. The software can be executed by one or more processing units of the processing subsystem 1204. The storage subsystem 1218 can also provide a repository for storing data used in accordance with the teachings of the present disclosure.

[0223] The storage subsystem 1218 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. Fig.12 As shown, the storage subsystem 1218 includes a system memory 1210 and a computer-readable storage medium 1222. The system memory 1210 may include multiple memories, including a volatile main random access memory (RAM) for storing instructions and data during program execution and a non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some implementations, a basic input / output system (BIOS) containing basic routines that help transfer information between elements within the computer system 1200, such as during startup, may typically be stored in ROM. The RAM typically contains data and / or program modules currently operated and executed by the processing subsystem 1204. In some implementations, the system memory 1210 may include multiple different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0224] By way of example and not limitation, Fig.12 As shown, system memory 1210 may load application programs 1212 being executed, which may include various applications (such as a web browser, a middle-tier application, a relational database management system (RDBMS)), program data 1214, and an operating system 1216. As an example, operating system 1216 may include various versions of Microsoft Apple and / or Linux operating system, various commercial or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google OS, etc.) and / or mobile operating systems such as Phone, OS, OS, OS operating system, and other operating systems.

[0225] Computer-readable storage media 1222 may store programming and data structures that provide the functionality of some embodiments. Computer-readable storage media 1222 may provide storage for computer-readable instructions, data structures, program modules, and other data for computer system 1200. When executed by processing subsystem 1204, the software (programs, code modules, instructions) that provide the above functionality may be stored in storage subsystem 1218. As an example, computer-readable storage media 1222 may include non-volatile memory, such as a hard drive, a disk drive, a hard drive such as a CD ROM, a DVD, (Blu-ray) disk or other optical media. Computer readable storage media 1222 may include, but is not limited to, Drive, flash memory card, universal serial bus (USB) flash drive, secure digital (SD) card, DVD disk, digital video tape, etc. The computer-readable storage medium 1222 may also include a solid-state drive (SSD) based on non-volatile memory (such as a flash memory-based SSD, an enterprise flash drive, a solid-state ROM, etc.), a volatile memory-based SSD (such as a solid-state RAM, dynamic RAM, static RAM, DRAM-based SSD, magnetoresistive RAM (MRAM) SSD), and a hybrid SSD using a combination of DRAM-based and flash memory-based SSDs.

[0226] In certain embodiments, storage subsystem 1218 may also include a computer-readable storage media reader 1220, which may also be connected to computer-readable storage media 1222. Computer-readable storage media reader 1220 may receive and be configured to read data from a memory device such as a disk, flash drive, or the like.

[0227] In some embodiments, computer system 1200 can support virtualization technology, including but not limited to virtualization of processing and memory resources. For example, computer system 1200 can provide support for executing one or more virtual machines. In some embodiments, computer system 1200 can execute programs such as hypervisors that facilitate the configuration and management of virtual machines. Memory, computing (e.g., processors, cores), I / O, and networking resources can be allocated to each virtual machine. Each virtual machine usually runs independently of other virtual machines. Virtual machines usually run their own operating system, which can be the same or different from the operating system executed by other virtual machines executed by computer system 1200. Therefore, computer system 1200 can potentially run multiple operating systems simultaneously.

[0228] The communication subsystem 1224 provides an interface to other computer systems and networks. The communication subsystem 1224 is used as an interface for receiving data from the computer system 1200 and transmitting data to other systems. For example, the communication subsystem 1224 can enable the computer system 1200 to establish a communication channel to one or more client devices via the Internet for receiving information from the client devices and sending information to the client devices.

[0229] The communication subsystem 1224 can support both wired and / or wireless communication protocols. For example, in some embodiments, the communication subsystem 1224 may include a radio frequency (RF) transceiver component, a global positioning system (GPS) receiver component, and / or other components for accessing a wireless voice and / or data network (e.g., using cellular telephone technology, advanced data network technology (such as 3G, 4G or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.XX family of standards), or other mobile communication technologies, or any combination thereof). In some embodiments, the communication subsystem 1224 may provide a wired network connection (e.g., Ethernet) in addition to or in lieu of a wireless interface.

[0230] The communication subsystem 1224 may receive and transmit data in a variety of forms. For example, in some embodiments, the communication subsystem 1224 may receive incoming communications in the form of structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc., among other forms. For example, the communication subsystem 1224 may be configured to receive, in real time, incoming communications from users of a social media network and / or a user of a website such as feed, The data feed 1226 is received (or sent) by other communication services such as web feeds, updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0231] In certain embodiments, the communication subsystem 1224 may be configured to receive data in the form of a continuous data stream that may be continuous or unbounded in nature and has no clear end, wherein the continuous data stream may include an event stream 1228 of real-time events and / or event updates 1230. Examples of applications that generate continuous data may include, for example, sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0232] The communication subsystem 1224 may also be configured to transmit data from the computer system 1200 to other computer systems or networks. The data may be transmitted in a variety of different forms, such as structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc. to one or more databases that may communicate with one or more streaming data sources coupled to the computer system 1200.

[0233] Computer system 1200 may be one of various types, including a handheld portable device (e.g., Cellular phone, computing tablets, PDAs), wearable devices (e.g., Google head mounted display), personal computer, workstation, mainframe, kiosk, server rack, or any other data processing system. Due to the ever-changing nature of computers and networks, Fig.12 The description of the computer system 1200 depicted in FIG. 1 is intended to be a specific example only. Fig.12 Many other configurations of more or fewer components of the system depicted are possible. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will understand other ways and / or methods to implement the various embodiments.

[0234] Although specific embodiments have been described, various modifications, changes, alternative constructions, and equivalents are possible. Embodiments are not limited to operations within certain specific data processing environments, but can be freely operated within multiple data processing environments. In addition, although certain embodiments have been described using a series of specific transactions and steps, it is clear to those skilled in the art that this is not intended to be limiting. Although some flow charts describe operations as sequential processing, many operations can be performed in parallel or simultaneously. In addition, the order of operations can be rearranged. Processing may have other steps not included in the figure. The various features and aspects of the above-described embodiments can be used alone or in combination.

[0235] In addition, although certain embodiments have been described using a particular combination of hardware and software, it should be appreciated that other combinations of hardware and software are also possible. Certain embodiments may be implemented using only hardware or only software or a combination thereof. The various processes described herein may be implemented on the same processor or on different processors in any combination.

[0236] Where a device, system, component or module is described as being configured to perform certain operations or functions, such configuration may be achieved, for example, by designing electronic circuits to perform the operations, by programming programmable electronic circuits (such as microprocessors) to perform the operations, such as by executing computer instructions or code, or a processor or core is programmed to execute code or instructions stored on a non-transitory storage medium, or any combination thereof to perform the operations. Processes may communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0237] Specific details are given in this disclosure to provide a thorough understanding of the embodiments. However, embodiments may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary details to avoid obscuring the embodiments. This description provides only example embodiments and is not intended to limit the scope, applicability, or configuration of other embodiments. On the contrary, the previous description of the embodiments will provide an enabling description for implementing various embodiments for those skilled in the art. Various changes may be made to the functions and arrangements of the elements.

[0238] Therefore, the description and drawings should be regarded as illustrative rather than restrictive. However, it will be clear that additions, subtractions, deletions and other modifications and changes may be made thereto without departing from the broader spirit and scope set forth in the claims. Therefore, although specific embodiments have been described, these embodiments are not intended to be limiting. Various modifications and equivalents are within the scope of the appended claims.

Claims

1. A computer-implemented method for generating a customized knowledge graph for an application, the computer-implemented method comprising: Receive a user data set of the application; Extract entities from user datasets; Identify links between entities based on user datasets; Creating a seed graph representing entities and links between entities, the seed graph comprising a set of nodes and a set of edges, each node in the set of nodes representing an entity extracted from a user data set, and each edge in the set of edges connecting two nodes and representing a link between the entities represented by the two nodes; identifying one or more weakly connected components in the seed graph, wherein each weakly connected component is a subgraph of the seed graph in which all nodes in the subgraph can be reached by traversing the edges of the subgraph along an undirected path; For each weakly connected component in the seed graph: Map entities in weakly connected components to vertices in the reference knowledge graph; Starting from the mapped vertex in the reference knowledge graph and traversing the reference knowledge graph based on a finite state machine to identify up to a first threshold number of entities in the reference knowledge graph; as well as Saving the Resource Description Framework RDF of the identified entity as an entry in the buffer; Calculate the priority scores of the entries in the buffer; Selecting from the entries in the buffer a first set of entries having the highest priority scores; as well as The entities and links identified by the first set of entries are added to the seed graph to generate a customized version of the knowledge graph.

2. The computer-implemented method of claim 1, wherein the user dataset comprises a plurality of user utterances.

3. The computer-implemented method of claim 1 , wherein extracting entities and identifying links between entities comprises performing at least one of the following on the user dataset: Part-of-speech tagging; Named Entity Recognition; or Composition analysis.

4. The computer-implemented method of claim 1, wherein mapping entities in the weakly connected components to vertices in the reference knowledge graph comprises mapping the entities to vertices in the reference knowledge graph using Dexter 2.

5. The computer-implemented method of claim 1 , wherein: The finite state machine includes disambiguation state, entity state, forbidden state and end state; as well as Traversing the reference knowledge graph includes: When the next vertex in the reference knowledge graph is a disambiguation vertex, it enters the disambiguation state; When the next vertex in the reference knowledge graph is a forbidden vertex: If the current state of the finite state machine is not the prohibited state, then enter the prohibited state; When the next vertex in the reference knowledge graph is an entity vertex: Entering a physical state; and saving the RDF of the next vertex in the reference knowledge graph as an entry in the buffer; and When the number of entries in the buffer is greater than a second threshold number, the end state is entered.

6. The computer-implemented method of claim 5, wherein forbidden vertices include dates, time values, links to another reference knowledge graph, web pages, or duplicate or redirected entities.

7. The computer-implemented method of claim 1 , wherein the priority scores of the entries in the buffer are determined using a priority function, the priority function being a function of: The number of weakly connected components in the knowledge graph that includes the seed graph and the entities associated with the entry; a graph density of a knowledge graph including a seed graph and entities associated with the entry; and Control parameters.

8. The computer-implemented method of claim 7, further comprising: After adding the entities and links identified in the first set of entries to the seed graph, the control parameters are updated.

9. The computer-implemented method of claim 8, wherein updating the control parameter comprises multiplying the control parameter by a value less than 1.

10. The computer-implemented method of claim 8, further comprising: For each weakly connected component in your customized version of the knowledge graph: Map entities in weakly connected components to vertices in the reference knowledge graph; Starting from the mapped vertex in the reference knowledge graph and traversing the reference knowledge graph based on a finite state machine to identify up to a first threshold number of entities in the reference knowledge graph; as well as Save the RDF of the identified entity as an entry in the buffer; calculating a priority score for an entry in the buffer using the priority function and the updated control parameter; selecting a second set of entries having the highest priority scores from among the entries in the buffer; as well as The entities and links identified by the second set of entries are added to the seed graph to generate an updated customized knowledge graph.

11. The computer-implemented method of claim 1 further comprising, before calculating the priority scores for the entries in the buffer, performing reservoir sampling on the entries in the buffer to reduce the number of entries in the buffer.

12. The computer-implemented method of claim 1 , further comprising: receiving input speech; Extract entities from input utterances; Generate knowledge graph embeddings for the extracted entities based on the customized knowledge graph; as well as Classify the input utterance based on the knowledge graph embedding of the extracted entities.

13. The computer-implemented method of claim 12, wherein generating the knowledge graph embedding comprises generating the knowledge graph embedding using TransE or an extension of TransE.

14. The computer-implemented method of claim 12, wherein generating a knowledge graph embedding comprises generating the knowledge graph embedding using TransE and a weighted scoring function.

15. The computer-implemented method of claim 12, wherein generating a knowledge graph embedding comprises: Pre-train knowledge graph embeddings of entities extracted from input utterances using a reference knowledge graph; as well as Retrain the knowledge graph embeddings of entities extracted from the input utterance using the custom knowledge graph and pre-trained knowledge graph embeddings.

16. The computer-implemented method of claim 12, wherein classifying the input utterance based on the knowledge graph embedding of the extracted entities comprises: Determine the average knowledge graph embedding of the knowledge graph embeddings of the extracted entities; determining the average GloVe vector of the GloVe vectors of the words in the input utterance; and Classify the input utterance based on the average knowledge graph embedding and the average GloVe vector.

17. The computer-implemented method of claim 16, wherein classifying the input utterance based on the average knowledge graph embedding and the average GloVe vector comprises: Combine the average knowledge graph embedding and the average GloVe vector to generate a concatenated vector; Perform principal component analysis on the concatenated vectors; as well as Classify input utterances based on principal component analysis.

18. The computer-implemented method of claim 16, wherein determining an average GloVe vector of GloVe vectors of words in an input utterance comprises: Clean input speech; Generate GloVe vectors for each word in the cleaned input utterance; as well as Average the GloVe vectors for each word in the cleaned input utterance.

19. A non-transitory computer readable medium storing a plurality of instructions executable by one or more processors, wherein the plurality of instructions, when executed by the one or more processors, cause the one or more processors to perform a process comprising: Receive a user data set of the application; Extract entities from user datasets; Identify links between entities based on user datasets; Creating a seed graph representing entities and links between entities, the seed graph comprising a set of nodes and a set of edges, each node in the set of nodes representing an entity extracted from a user data set, and each edge in the set of edges connecting two nodes and representing a link between the entities represented by the two nodes; identifying one or more weakly connected components in the seed graph, wherein each weakly connected component is a subgraph of the seed graph in which all nodes in the subgraph can be reached by traversing the edges of the subgraph along an undirected path; For each weakly connected component in the seed graph: Map entities in weakly connected components to vertices in the reference knowledge graph; Starting from the mapped vertex in the reference knowledge graph and traversing the reference knowledge graph based on a finite state machine to identify up to a first threshold number of entities in the reference knowledge graph; as well as Saving the Resource Description Framework RDF of the identified entity as an entry in the buffer; Calculate the priority scores of the entries in the buffer; Selecting from the entries in the buffer a first set of entries having the highest priority scores; as well as The entities and links identified in the first set of entries are added to the seed graph to generate a customized version of the knowledge graph.

20. A system for generating a customized knowledge graph for an application, the system comprising: one or more processors; as well as A memory coupled to the one or more processors, the memory storing instructions that, when executed by the one or more processors, cause the system to perform operations including: Receive a user data set of the application; Extract entities from user datasets; Identify links between entities based on user datasets; Creating a seed graph representing entities and links between entities, the seed graph comprising a set of nodes and a set of edges, each node in the set of nodes representing an entity extracted from a user data set, and each edge in the set of edges connecting two nodes and representing a link between the entities represented by the two nodes; identifying one or more weakly connected components in the seed graph, wherein each weakly connected component is a subgraph of the seed graph in which all nodes in the subgraph can be reached by traversing the edges of the subgraph along an undirected path; For each weakly connected component in the seed graph: Map entities in weakly connected components to vertices in the reference knowledge graph; Starting from the mapped vertex in the reference knowledge graph and traversing the reference knowledge graph based on a finite state machine to identify up to a first threshold number of entities in the reference knowledge graph; as well as Saving the Resource Description Framework RDF of the identified entity as an entry in the buffer; Calculate the priority scores of the entries in the buffer; Selecting from the entries in the buffer a first set of entries having the highest priority scores; as well as The entities and links identified in the first set of entries are added to the seed graph to generate a customized version of the knowledge graph.

Images