Knowledge graph assisted large language model

Abstract

Techniques for a knowledge graph system use a large language model (LLM) to construct a knowledge graph to answer queries submitted by a user to a chat robot. The knowledge graph system constructs the knowledge graph using answers generated by LLM for a new query. The chat robot will continue to use the LLM to answer new queries, but the chat robot may utilize the knowledge graph to answer duplicate questions to obtain various efficiencies on LLM-supported chat robots. For example, the knowledge graph system may easily debug or otherwise improve the answers in a knowledge graph, store provenance information in a knowledge graph, and enhance the knowledge graph using other data sources. Thus, the reliability and correctness of the chat robot will be improved because errors and inaccuracies in answers provided by the LLM will be corrected in the knowledge graph, but the chat robot can still take advantage of the LLM's ability to provide answers over various subject domains.

Description

[0001] Cross-references to related applications This application claims priority to U.S. Patent Application No. 18 / 478,702, filed September 29, 2023, entitled “Knowledge Graph Assisted Large Language Models” and U.S. Patent Application No. 18 / 375,256, filed September 29, 2023, entitled “Generating Knowledge Graphs Using Large Language Models”, the entire contents of which are incorporated herein by reference. Background Technology

[0002] Various types of virtual agents have emerged over the years, designed to interact with and assist users as if they were human assistants. One type of virtual agent, called a chatbot, is a computer program that converses with users via text or voice. Traditionally, chatbots operate under rule-based systems, where rules and decision trees are used to identify specific words or phrases provided by the user and to provide predefined responses based on those words or phrases. However, these chatbots are quite limited and struggle to handle unexpected or complex queries from users. Therefore, while rule-based chatbots can handle basic tasks, they have rather limited usefulness and offer little value to users.

[0003] Recent advancements in artificial intelligence (AI) have enabled chatbots and other AI systems to perform complex tasks that typically require artificial intelligence. Generative AI is a type of AI where models are used to create (or “generate”) new content based on input, often in the form of prompts from the user. One type of generative AI model is particularly effective at generating text, especially Large Language Models (LLMs). LLMs are trained on large collections or corpora of text data to perceive and infer the context of user queries, understand a wider range of queries, and generate human-like textual responses to queries. Chatbots powered by LLMs are becoming increasingly popular among users due to their ability to perform complex tasks on behalf of users. Attached Figure Description

[0004] The following detailed description is given with reference to the accompanying drawings. In the drawings, the leftmost numeral indicates the first figure in which the numeral appears. The same numerals are used in different figures to indicate similar or identical items. The systems depicted in the drawings are not drawn to scale, and the components in the drawings may not be depicted to scale with each other.

[0005] Figure 1 An example system architecture diagram is shown, illustrating a scenario where a knowledge graph system uses LLM to generate knowledge graphs that a chatbot system uses to answer user queries.

[0006] Figure 2 A component diagram is shown, illustrating an example component of a service provider system that uses an LLM-generated knowledge graph system.

[0007] Figure 3 An example timeline flow is illustrated, according to which queries are processed through a pipeline that includes the use of knowledge graphs and LLM to determine the answer to a query.

[0008] Figure 4A This example illustrates a chatbot system's interface, which presents origin data of the responses to queries and prompts the user to report any errors in the responses.

[0009] Figure 4B An example system architecture diagram is shown, in which a chatbot system provides user feedback for answers and a knowledge graph system modifies the knowledge graph used to determine the answer based on the feedback.

[0010] Figure 5 An example system architecture diagram with a sample interface is provided, which prompts engineering components to provide relevant portions of the knowledge graph to the LLM to help determine the answer to the query.

[0011] Figure 6 This example demonstrates a context window interface where the prompting engineering component generates a summary prompt suitable for an LLM-supported chatbot by summarizing the initial prompt.

[0012] Figure 7 A flowchart illustrating an example process in which a knowledge graph system uses LLM to determine the answer to a query and adds the answer to the query to the knowledge graph is provided.

[0013] Figure 8 A flowchart illustrating an example process for a knowledge graph system to iteratively prompt an LLM to determine answers to multiple queries and to construct the knowledge graph by adding the answers to the queries to the knowledge graph.

[0014] Figure 9 The flowchart illustrates an example process where a chatbot receives a query in natural language, translates it into query language, and determines that the knowledge graph does not have an answer to the query. The chatbot then uses LLM to determine the answer, adds it to the knowledge graph, and provides the answer to the user.

[0015] Figure 10A and Figure 10BThe flowchart illustrates an example process for constructing a knowledge graph using answers from an LLM, receiving a query from a user, and then determining whether the knowledge graph or the LLM can answer the query.

[0016] Figure 11 This document illustrates a flowchart of an example process where a chatbot receives a query from a user's device, uses an LLM to determine the response to the query, and adds the response to the query to a knowledge graph for future use.

[0017] Figure 12 This is a system and network diagram illustrating an exemplary operating environment of a data center that includes aspects of a service provider system that can be configured to implement the functionality described herein.

[0018] Figure 13 This is a computing system diagram illustrating the configuration of a data center that can be used to implement various aspects of the techniques disclosed herein.

[0019] Figure 14 This is a computer architecture diagram illustrating an exemplary computer hardware architecture for implementing various aspects of the computing devices that can be used to implement the various technologies presented herein. Detailed Implementation

[0020] This disclosure describes a technique for knowledge graph systems that uses Large Language Models (LLMs) to construct knowledge graphs to answer queries submitted by users to a chatbot. Recent advancements in generative AI, particularly around chatbots, code generators (and other domain-specific LLMs), and similar models and services that use LLMs to generate text, have been noteworthy. However, these LLM-based systems suffer from various limitations and drawbacks, such as the time and resource consumption of fine-tuning LLMs, the generation of incorrect or misleading answers by LLMs, and the difficulty in determining the source of the answers generated by LLMs. According to the technique described herein, the knowledge graph system uses answers generated by LLMs to construct a knowledge graph, allowing the knowledge graph to grow whenever the LLM answers a new query. The chatbot continues to use the LLM to answer new queries, but the chatbot can leverage the knowledge graph to answer recurring questions, achieving various efficiencies on an LLM-enabled chatbot. For example, the knowledge graph system can easily debug or otherwise improve the answers in the knowledge graph, store origin information for the answers in the knowledge graph, and use other data sources to enhance the knowledge graph. In this way, knowledge graphs can be built by leveraging the knowledge and capabilities of LLM, but the various drawbacks of LLM can be avoided using knowledge graphs. Therefore, the reliability and accuracy of the chatbot will be improved because errors and inaccuracies in the answers provided by the LLM will be corrected in the knowledge graph, while the chatbot can still utilize the ability of LLM to provide answers across various subject domains.

[0021] As described above, a chatbot is a computer program designed to converse with a user via text or voice. Chatbots can use various types of models or algorithms to determine responses to user queries, such as rule-based models, machine learning (ML) models, natural language processing (NLP) models, AI models, and knowledge graphs. Typically, knowledge graphs are data structures used to organize information and represent knowledge in a structured format. As their name suggests, knowledge graphs can be graph-like data structures with nodes, edges, attributes, and labels. Nodes in the graph represent entities or concepts (e.g., person locations, objects, etc.), edges represent relationships between nodes, attributes provide additional metadata or information about the entities or relationships, and labels can be attached to nodes and edges to indicate the kind of entity or relationship being represented.

[0022] Chatbots powered by knowledge graphs can analyze and understand the rules, entities, and relationships within the knowledge graph and use this information to answer user questions. Due to the relatively straightforward rules and structure of knowledge graphs, they are quite easy to build, modify, and enhance. For example, answers generated by a knowledge graph can easily be rooted to specific parts of the knowledge graph to explain why a particular answer to a query was derived. This is especially helpful for incorrect, confusing, or inaccurate answers. Furthermore, knowledge graphs can be immediately changed or modified to fix those errors, and the downstream effects of the modifications are easily identifiable and predictable. Therefore, knowledge graphs provide an inexpensive mechanism for instantly updating, fixing, and changing chatbots. However, building knowledge graphs often requires large datasets that may be difficult to obtain and are typically dedicated to a specific topic domain. Additionally, knowledge graph-powered chatbots can be quite limited and struggle with handling unexpected or complex queries from users.

[0023] Conversely, LLMs are extremely efficient at understanding and processing complex queries from users and are also capable of responding to problems in arbitrary domains. While LLMs are also trained on large amounts of data, there are many pre-trained and readily available LLMs that do not require additional data for training. Furthermore, the algorithms used to train LLMs enable them to perceive and infer the context of user queries, understand a wider range of queries, and generate human-like textual responses to queries. LLM-powered chatbots have become increasingly popular due to their ability to understand complex queries and perform complex tasks on behalf of users. LLMs are particularly useful in processing and understanding queries that are novel or not readily apparent to the LLM, and in providing answers to those queries. However, as described above, these LLM-based chatbots have various limitations and drawbacks, such as the time-consuming nature of fine-tuning LLMs, the LLM "illusion" of producing incorrect or misleading answers, and the difficulty in determining the source of the answers generated by the LLM.

[0024] According to the techniques described herein, chatbots can answer user queries by working in conjunction with a knowledge graph system that at least partially utilizes responses from an LLM (Local Language Graph) to generate, construct, or enhance the knowledge graph. When a chatbot receives a query from a user, the knowledge graph system can initially attempt to use the knowledge graph to determine an answer to the query. Queries are typically written or expressed by the user in natural language, and the knowledge graph system may include a query evaluation engine that translates the query from natural language into a query language such as Resource Description Framework (RDF) query language or SPARQL. In some instances, the query engine may leverage a translation engine to translate the query into query language. The query engine can then use the knowledge graph to evaluate the query and determine whether the knowledge graph returns a result. In examples where the knowledge graph does return a result or "answer" to the query, the chatbot can simply provide the user with the answer returned by the knowledge graph.

[0025] However, in some examples, the knowledge graph does not return any results for the query. In such examples, the knowledge graph system can determine and prompt the LLM with an answer to the query. Depending on the LLM, the query can be submitted in natural language form, query language form, and / or a combination thereof. The knowledge graph system can request the LLM to provide a true answer to the query, and the LLM can output its answer to the query to the knowledge graph system.

[0026] In some cases, a knowledge graph system can simply add the answer to the knowledge graph as a response to a query and provide the answer to a chatbot for use in responding to users. However, in other instances, a knowledge graph system can perform validation techniques to determine if the answer is accurate for the query. For example, the knowledge graph system can request one or more auxiliary LLMs to confirm that they agree the answer is accurate for the query. As another example, a knowledge graph system can query remote sources of data, such as internet-based repositories, to confirm the accuracy of the answer to the query. In this way, the knowledge graph system can add the answer to the knowledge graph and build a knowledge base for the knowledge graph.

[0027] In some examples, chatbots can provide users with various options and information. As an example, a chatbot might be able to provide origin data for answers to queries. Origin data can indicate the origin or source of the answer, such as the Uniform Resource Locator (URL) of the data source, the name of the entity providing the answer, etc. A knowledge graph system can store origin information in the knowledge graph and map it to answers. In this way, if a user requests the chatbot to provide an indication of the source of an answer, the chatbot can work with the knowledge graph system to obtain the origin information of the answer (or multiple answers) and provide that information to the user. The user can then decide whether they trust the source of the answer and follow up to verify the answer themselves.

[0028] In some instances, chatbots can provide feedback mechanisms that allow them to offer feedback on the accuracy of the answers they provide. As a simple example, users might be able to highlight a portion of the answer or simply flag the entire answer as incorrect or confusing. The knowledge graph system and / or the administrator associated with it can determine whether the answer is accurate, and if so, modify or remove it.

[0029] While the techniques described herein are based on building knowledge graphs while operating a chatbot, these techniques are equally applicable to simply building or enhancing knowledge graphs without queries from a chatbot. For example, a knowledge graph system can simply take multiple questions, either related to a general topic or specific to the subject domain where the knowledge graph will be used. The knowledge graph system can then iteratively prompt an LLM to provide answers to the multiple questions and use those answers to build or argue the knowledge graph. Thus, a knowledge graph system can act as or represent a graph-as-a-service, where it is potentially able to generate knowledge graphs from scratch using one or more LLMs. This can be advantageous because generating knowledge graphs typically requires large datasets and significant time. Additionally, a knowledge graph system can be able to improve or enhance organized domain-specific knowledge graphs by leveraging the prompting engineering of an LLM to add general knowledge to these knowledge graphs.

[0030] LLMs do indeed have a “context window,” which typically represents the amount of information an LLM can consider when processing new input to generate content, analogous to human short-term memory. The larger the context window, the more data can be input into the LLM and analyzed by it to generate a response. Therefore, while it might seem intuitive for a knowledge graph system to provide the LLM with all the information available for a question, the amount of information may exceed the size of the LLM's context window. For example, a knowledge graph system might want to provide the LLM with the entire knowledge graph for context and use it when generating an answer to a query. However, the amount of data in the knowledge graph may exceed the size of the LLM's context window.

[0031] To handle context window limitations, a knowledge graph system can provide the LLM with a portion of the knowledge graph, such as the most relevant information for the query. For example, the knowledge graph system can determine which part of the knowledge graph is semantically most relevant, or has an answer relevant to the query. The knowledge graph system can then provide the LLM with the most relevant information along with the query. Furthermore, instead of providing relevant information in the language or structure of the knowledge graph (e.g., semantic triples, RDF triples, etc.), the knowledge graph system can generate a summary of the relevant information (potentially leveraging the LLM) to reduce the amount of data provided to the LLM.

[0032] Language models (e.g., large language models, small language models, etc.) are a type of artificial intelligence (AI) model trained on text data to generate coherent and context-sensitive text. "Large" language models refer to those that have been trained on massive datasets and have a large number of parameters, enabling them to capture complex language patterns and perform a wider range of tasks. Large language models are designed to handle a broad range of natural language processing tasks, such as text completion, translation, summarization, and even dialogue. The specific number of parameters required for a model to be considered a "large" language model can vary depending on context and technological advancements. However, traditionally, large language models have millions to billions of parameters.

[0033] While some of the techniques described in this paper reference large language models, these techniques are equally applicable to any type of language model, such as small language models, domain-specific language models, transformer-based models, unified language models, and other types of language models.

[0034] Certain specific embodiments and implementations of this disclosure will now be described more fully below with reference to the accompanying drawings, in which various aspects are illustrated. However, these aspects may be implemented in many different forms and should not be construed as limited to the specific embodiments set forth herein. This disclosure covers variations of the embodiments described herein. The same reference numerals denote the same elements throughout.

[0035] Figure 1 This example illustrates the system architecture diagram of an environment 100 in which the knowledge graph system 104 of the service provider system 102 uses LLM to generate a knowledge graph that is used by the chatbot system 110 to answer user queries.

[0036] Service provider system 102 may be operated and / or managed by a service provider and may provide one or more services to users. As described in more detail below, service provider system 102 may be or include cloud systems, such as cloud-based computing resources, that provide services and resources to meet its computing resource needs. In other examples, service provider system 102 may simply support and provide chatbot system 110, which includes hardware (e.g., computing devices, network connectivity, etc.) and computer programs designed to converse with users via text or voice. In some instances, service provider system 102 may be a private network, such as a corporate network, an on-premises network, etc.

[0037] Chatbot system 110 may use the knowledge graph and / or LLM described herein to determine the response to a query from user 106. Chatbot system 110 may include a chatbot component 112 that acts as the commander of chatbot system 110 and communicates with other components, and one or more chatbot interfaces 114 that provide a mechanism through which user 106 can submit queries. Chatbot interface 114 may be a chat interface through which user 106 and / or the program can submit text (and other input) prompts. However, chatbot interface 114 may be any type of interface through which software and / or user 106 can communicate with chatbot system 110, such as CLI, API, or any other type of software instruction.

[0038] As shown in the figure, user 106 can submit a query to chatbot user interface (UI) 124 at point 1 using user device 108. This query can be input as a natural language query 126 in plain language or natural language, such as human language used for communication. At point 2, chatbot interface 114 can receive the natural language query 126, and chatbot component 112 can translate the natural language query 126 into a formal language query 128. The formal language can be any type of language that can be used to query knowledge graph 122. For example, the formal language can be Structured Query Language (SQL), the SPARQL protocol and RDF Query Language (SPARQL), the Resource Description Framework (RDF) language, or any other formal query language. Chatbot component 112 can use various models or algorithms to convert the natural language query 126 into the formal language query 128, such as one or more LLMs configured to translate between languages.

[0039] Chatbot system 110 may provide formal language query 128 to knowledge graph system 104. Therefore, chatbot system 110 can answer user 106's query by working in conjunction with knowledge graph system 104, which may at least partially utilize the answer from LLM 118 of AI system 116 to generate, construct, or enhance knowledge graph 122. When chatbot system 110 receives query 128 from user 106, knowledge graph system 104 may initially attempt to use knowledge graph 122 to determine an answer 134 to query 128. Knowledge graph component 120 may include a query evaluation engine that uses knowledge graph 122 to evaluate formal language query 128 to determine whether knowledge graph 122 returns a result (e.g., an answer) for a particular query. In the example where knowledge graph 122 does indeed return the result or formal language answer 134 to formal language query 128, chatbot system 110 can simply convert formal language answer 134 into natural language answer 136 and provide user 106 with the natural language answer 136 determined using knowledge graph 122.

[0040] However, in some examples, knowledge graph 122 does not return any results for formal language query 128. In such examples, knowledge graph component 120 may determine a hint 132 at point 4 to submit an answer 134 to query 128 to LLM 118. Depending on LLM 118, query 128 may be submitted in natural language form, formal language form, and / or a combination thereof. Knowledge graph system 104 may request LLM 118 to provide a true answer 134 to query 128, and LLM 118 may output a formal language answer 134 to query 128 to knowledge graph component 120 at point 5.

[0041] In some instances, knowledge graph system 104 may simply add formal language answer 134 as an answer to formal language query 128 to knowledge graph 122 at point 6 and provide formal language answer 134 to chatbot system 110 for responding to user 106. However, in some instances, knowledge graph system 104 may perform deterministic techniques to determine whether formal language answer 134 is accurate for formal language query 128. For example, knowledge graph system 104 may request one or more auxiliary LLMs (not illustrated) to confirm that they agree that formal language answer 134 is accurate for formal language query 128. As another example, knowledge graph system 104 may query remote data sources, such as internet-based repositories, to confirm that formal language answer 134 is accurate for formal language query 128. In this way, knowledge graph component 120 may add the answer to knowledge graph 122 and construct a knowledge base for knowledge graph 122. At point 7, the chatbot system 110 may provide a natural language response 136 to the user 106 for presentation on the chatbot UI 124. In some instances, the response may be at least partially the natural language response 136 presented to the user 106.

[0042] While the techniques described herein are based on the construction of a knowledge graph 122 while operating a chatbot system 110, these techniques are equally applicable to simply constructing or enhancing a knowledge graph 122 without requiring formal language queries 128 via the chatbot system 110. For example, the knowledge graph system 104 may simply obtain multiple questions, which may be related to a general topic or specific to the topic domain where the knowledge graph 122 will be used. The knowledge graph system 104 may then iteratively prompt the LLM 118 to provide formal language answers 134 to the multiple questions and use the answers 134 to construct or argue the knowledge graph 122. As another example, the knowledge graph component 120 may select a topic region and require the LLM 118 to iteratively and repeatedly explore concepts in that space and use these questions and answers to construct the knowledge graph 122. Thus, the knowledge graph system 104 may act as or represent a graph service, where it is potentially able to generate the knowledge graph 122 from scratch using one or more LLMs 118. This can be advantageous because typically large datasets and significant time are required to generate the knowledge graph 122. Additionally, the knowledge graph system 104 can improve or enhance the organized domain-specific knowledge graphs by adding general knowledge to these knowledge graphs 122 using the hinting engineering of the LLM 118.

[0043] AI systems 116 provide artificial intelligence, which generally refers to the ability of computers to perform tasks typically requiring artificial intelligence, such as perceiving, synthesizing, and inferring information. Generally, AI systems 116 and models ingest large amounts of data (or "training data"), analyze this data to identify correlations and patterns, and use these patterns to make predictions about future states. Although AI programs and algorithms have existed for decades, the amount of data and computing power required to train AI models useful to humans does not yet exist. However, various technological breakthroughs and advancements have occurred that accelerate the usefulness of AI, such as the emergence of cloud computing providing efficient, unlimited computing power, advancements in dedicated hardware (e.g., graphics processing units (GPUs)) for effectively training and running these AI models, and the discovery of more efficient training algorithms.

[0044] One type of artificial intelligence that has greatly benefited from these advancements is generative AI. Generative AI is a form of artificial intelligence in which AI models are used to create (or “generate”) new content based on input, often in the form of prompts from a human. Various types of generative AI models exist, trained to generate different types of data or content, such as text, images, audio (e.g., music or speech), and synthetic or other virtual data. Similar to other forms of AI, generative AI models use machine learning (ML) techniques to learn how to create content that exhibits characteristics learned from the training data used to train the model. More specifically, generative AI models use neural networks (and / or other ML algorithms) to learn the underlying correlations, patterns, and structure of a dataset and generate new content.

[0045] A type of neural network architecture that has gained popularity due to its ability to reduce the amount of time required to train generative AI models is called a Transformer model, or simply "Transformer". Transformers apply a set of mathematical techniques called attention, or self-attention, to capture relationships in sequential data called tokens, such as words in a sentence. Transformers are able to detect subtle causal relationships between a range of data elements, including how distant data elements influence and depend on each other. Unlike previous models (e.g., recurrent neural networks (RNNs)) that had to process tokens sequentially, transformers use attention mechanisms to process tokens simultaneously and compute attention weights, or the strength of relationships, between tokens in successive layers. Because transformers can compute attention weights for all tokens in parallel, the amount of time required to train generative AI models using transformers is significantly reduced compared to other trained models.

[0046] Generative AI can be used to generate text that resembles human-like responses to prompts. Transformers are highly efficient in training models for generating text (often called Large Language Models (LLM 118)). LLM 118 is trained on large collections or corpora of text data to generate human-like textual responses to prompts. LLM 118 is typically trained in two phases: pre-training and fine-tuning. During the pre-training phase, LLM 118 is trained (or “unsupervised learning”) on a large dataset of unlabeled text data, where the transformer allows LLM 118 to process and learn patterns and relationships between words. In the fine-tuning phase, LLM 118 can be fine-tuned for specific tasks or prompts, such as summarizing content, answering questions, and text completion. There are general-purpose LLM 118s that have been trained on text datasets describing all types of content (e.g., data obtained from web crawlers scraping the public internet). There are also specialized LLM 118s that have been trained on specialized datasets dedicated to specific types of content, such as travel or shopping.

[0047] Numerous advancements have been made in large-scale machine learning and deep learning models. For example, Generative Pre-trained Model Version 3 (GPT-3) was trained on 570 GB of text and included 175 billion parameters. While large models may offer state-of-the-art performance, smaller models may be desirable in various situations. Knowledge distillation is a technique that transfers knowledge from a complex neural network (“teacher model”) to a simpler neural network (“student model”). The teacher model is trained with labeled data, and the student model is trained with unlabeled data containing a “soft target”—a probability distribution indicating the teacher’s confidence in its predictions—to mimic the teacher’s behavior. By minimizing the difference between the student’s predictions and the teacher’s soft target, the student model learns from the teacher’s knowledge and can achieve similar or better performance even with fewer parameters.

[0048] Organizations continue to train and use, or provide, generative AI models to perform various tasks. To interact with a text-based generative AI model, or LLM 118, a text box is presented to user 106, through which the user can submit prompts representing the agent of the LLM 118 for analysis and response. In addition to prompts or instructions, user 106 can also provide the agent with additional data for analysis. For example, user 106 can request the generative AI agent to analyze a dataset uploaded or otherwise input into the agent's LLM 118. The generative AI agent can then utilize the LLM 118 to analyze the uploaded data and perform tasks such as text summarization or data synthesis. These generative AI models have a "context window," which typically represents the amount of information the AI ​​model can consider when processing new input to generate content, similar to human memory. The larger the context window, the greater the amount of data that can be input into the AI ​​model and analyzed by the AI ​​model to generate a response. Therefore, while providing an AI model with all the information available for the question may seem intuitive to the user, the amount of information may exceed the AI ​​model's context window. In such cases, when prompting the AI ​​model to perform a task, users can instead choose to specify that only the most relevant information from the AI ​​model is provided.

[0049] Typically, LLM 118 can only parse a limited number of tokens (treating a token as approximately half a word, or about ~4 characters). For example, the LLM 118 in use might have a context window of 4096 tokens, which is approximately 4096 * 4 characters (the exact token-character mapping depends on the specific text provided). A key engineering challenge in working with the current generation of LLM 118 is managing this limited context window. In this example, the combination of chat history and responses from the LLM 118 must total at most 4096 tokens. If too many tokens are provided in the chat history, the API will fail. A language model's context window refers to the range of text a language model considers when processing or generating a specific word or token within a given text sequence. It represents the surrounding words or tokens the model uses to understand the context of the current word or token. The size of this context window is determined by the language model's architecture. Context windows help language models generate coherent and context-accurate responses in various natural language processing tasks, such as text completion, translation, question answering, etc. It allows the model to consider a wider range of context and the semantic meaning of words, ensuring its output matches the expected meaning of the input text. The size of this window varies depending on the model but typically includes both preceding and following words in the text sequence. Larger windows can accommodate additional information, such as relevant information added via Retrieval Augmentation Generation (RAG). However, it is important to note that larger context windows can be computationally expensive, and therefore the size of the context window can vary depending on the specific language model architecture and resource constraints.

[0050] User 106 may have already created a user account with Service Provider System 102 to utilize the resources of Service Provider System 102. User 106 may communicate with Service Provider System 102 via one or more networks 130 (e.g., WAN, PAN, LAN, etc.) using their user device 108. User device may include any type of computing device configured to communicate via network 130, such as mobile phone, tablet computer, laptop computer, desktop computer, television, server, and / or any other type of computing device. User 106 may interact with Service Provider System 102 via their user account and / or one or more user portals or consoles (e.g., web console, CLI, API, etc.).

[0051] Typically, the knowledge graph system 104 and its components may include software, firmware, and / or other logic supported by a computing device within the service provider system 102 or across multiple computing devices. Additionally, the knowledge graph system 104 may include systems of other devices. The techniques described herein are generally described in relation to the service provider system 102, such as a cloud provider network or system. However, these techniques are generally applicable to any network, such as on-premises networks, hybrid networks, etc.

[0052] As described above, service provider system 102 may be or include a cloud system that provides services and resources to meet the computing resource needs (such as cloud-based computing resources) of user 106. For example, user 106 may operate user device 108 to register for the use of computing resources of service provider system 102. Service provider system 102 may include one or more managed services, which may include components that provide different types of automated or semi-automated services to users, such as knowledge graph system 104. Typically, knowledge graph system 104 may be at least in part a control plane system that controls the operations occurring within service provider system 102. Knowledge graph system 104 may be centralized or distributed and supported by one or more computing devices.

[0053] In some instances, the knowledge graph system 104 and chatbot system 110 may be provided as plugins for various applications, browsers, web-based services, web-based interfaces, and / or software agents. In some examples, the knowledge graph system 104 and chatbot system 110 may actually run locally on the user's device and as applications on that device. The knowledge graph system 104 and chatbot system 110 may be packaged in a sufficiently small software application to run locally on the client device. In some instances, the code for the knowledge graph system 104 and chatbot system 110 may be open source and reside locally on the user's device to provide the various functionalities described herein. The knowledge graph system 104 and chatbot system 110 may manifest as an LLM-based chatbot that interfaces with the user 106 and runs locally on the device to provide various types of functionality (e.g., answering queries and performing tasks such as scheduling and sorting items on behalf of the user).

[0054] As shown in the figure, the service provider system 102 can be operated and / or managed by a service provider such as a cloud provider. The service provider system 102 can offer users various services to meet their computing resource needs, such as cloud-based computing resources. For example, the service provider system 102 can provide cloud-based, scalable, and network-accessible computing power services, storage services, database services, and / or other services. Users 106 can utilize computing devices to subscribe to the use of computing resources and / or services provided by the service provider system 102.

[0055] Service provider system 102 (often referred to as a cloud provider network or simply the "cloud") refers to a pool of network-accessible computing resources, such as computing, storage, and networking resources, applications, and services, which can be virtualized or bare metal. The cloud provides convenient, on-demand network access to a shared pool of configurable computing resources that can be programmatically configured and released in response to user commands. These resources can be dynamically provisioned and reconfigured to suit variable loads. Therefore, cloud computing can be viewed as applications delivered as a service over a publicly accessible network (e.g., the internet, cellular networks) and the hardware and software in the cloud provider's data center that provides those services.

[0056] Service provider system 102 can be structured into multiple zones, one of which is a separate geographical area where the cloud provider clusters its data centers. Each zone may include two or more availability zones interconnected via a private high-speed network (e.g., fiber optic communication connection). An availability zone (also called an availability domain, or simply a "zone") refers to an isolated fault domain comprising one or more data center facilities that have separate power, separate networking, and separate cooling from data center facilities in another availability zone. A data center refers to a physical building or rack that houses and provides power and cooling to the servers of the cloud provider's network. Preferably, availability zones within a zone are positioned far enough apart that the same natural disaster should not simultaneously take up more than one availability zone. Users can connect to the availability zones of the cloud provider's network via a transit center (TC) through a publicly accessible network (e.g., the Internet, cellular communication network). A TC can be considered the primary backbone location linking customers to the cloud provider's network and can be co-located at other network provider facilities (e.g., Internet service providers, telecommunications providers) and securely connected (e.g., via VPN or direct connection) to availability zones. Two or more TCs can operate in each zone for redundancy. The districts are connected to a global network that includes private network infrastructure (e.g., fiber optic connections controlled by a cloud provider) that connects each district to at least one other district. Service provider system 102 can deliver content from points of presence outside these districts but networked with them via edge locations and regional edge caching servers. This partitioning and geographical distribution of computing hardware enables service provider system 102 to provide customers with low-latency resource access globally with high fault tolerance and stability.

[0057] Cloud computing, rather than purchasing, owning, and maintaining their own data centers and servers, provides organizations with on-demand access to services such as computing power, storage, databases, and others. Service provider system 102 can provide organizations with on-demand, scalable computing services over a network, allowing them to have scalable “virtual computing devices” that they can handle, through their use of computing servers and block storage servers. These virtual computing devices possess the attributes of personal computing devices, including hardware (various types of processors, local memory, random access memory (“RAM”), hard disk and / or solid-state drive (“SSD”) storage devices), operating system selection, networking capabilities, and pre-loaded application software. Each virtual computing device can also virtualize its console input and output (“I / O”) (e.g., keyboard, monitor, and mouse). This virtualization allows organizations to connect to their virtual computing devices using computer applications (such as browsers, application programming interfaces, software development kits, etc.) to configure and use them as if they were personal computing devices. Unlike personal computing devices, which have a fixed number of hardware resources available to an organization, the hardware associated with virtual computing devices can be scaled up or down depending on the resources the organization needs. Organizations may choose to deploy their virtual computing systems to provide web-based services for their own use and / or for use by their users or clients.

[0058] Service provider system 102 can provide various computing resources or services, which may include virtual computing services, data processing services (e.g., map zooming, data streaming and / or other large-scale data processing technologies), data storage services (e.g., object storage services, block-based storage services, or data warehouse storage services), and / or any other type of network-based service (which may include various other types of storage, processing, analytics, communication, event processing, visualization, and security services not listed). The resources required to support the operation of such services (e.g., computing and storage resources) may be provided in an account associated with a cloud provider, as opposed to the resources requested by the organization of service provider system 102, which may be provided in a user account.

[0059] Service provider system 102 can offer organizations a variety of built-in services to help run their applications and services. For example, service provider system 102 can provide organizations with access to a Virtual Private Cloud (VPC), a logically isolated portion of service provider system 102 that serves as a private virtual environment accessible only to authorized accounts. An organization may have multiple VPCs, potentially spanning different areas of service provider system 102. To help interconnect VPCs and other resources of the user's computing infrastructure, the service provider can also offer a variety of built-in networking services. VPCs can be used by organizations to create subnets, configure personalized routing tables, select IP addresses, monitor connectivity, filter network traffic, restrict access to VPCs, and / or for other operations, as illustrated. VPCs include or run computing resources such as servers, virtual machines (VMs), containers, serverless functions, workloads, processes, etc. Typically, computing resources are used to support or run an organization's applications or services.

[0060] A Virtual Private Cloud (VPC) (also known in various implementations as a Virtual Network (VNet), Virtual Private Network, or Virtual Cloud Network) is a custom virtual network within another network, such as a cloud provider's network. A VPC can provide an underlying network layer for cloud services (e.g., compute cloud or edge cloud) or for customer applications or workloads running in the cloud. A VPC can be defined at least by its address space, internal structure (e.g., the compute resources and security groups that include the VPC), and transit paths.

[0061] A VPC is a virtual network dedicated to a specific customer account (or a collection of related customer accounts, such as different customer accounts belonging to the same business organization). A VPC is logically isolated from other virtual networks in the cloud. Customers can deploy resources such as compute instances into a VPC. When creating a VPC, a customer can specify a range of IPv4 addresses for the VPC in the form of Classless Inter-Domain Routing (CIDR) blocks. A VPC can span all availability zones within a specific region. After creating a VPC, a customer can add one or more subnets within each availability zone or at the edge location.

[0062] User 106 wants to deploy and host applications in a cloud system and wants to maintain privacy as if these applications were running on an on-premises network. Therefore, User 106 runs their applications in a VPC and on the computing resources of Service Provider System 102. Computing resources can include any type of resources, such as CPU resources (e.g., server processors, virtual machines, containers, etc.), GPU resources, memory resources (e.g., random access memory, cache memory, etc.), storage devices (e.g., random access memory (ROM), etc.), and networking resources.

[0063] Figure 2Component diagram 200 illustrates example components of a service provider system 102, including a knowledge graph system 104 that uses LLM 118 to generate a knowledge graph 122. As shown, the service provider system 102 may include a chatbot system 110 having a chatbot interface that can be used by a user 106 to communicate with the chatbot system 110, such as a chat interface (e.g., a user interface that allows a user to interact with a system or application using natural language dialogue), an API (e.g., a set of rules and protocols that define how different software components can interact and communicate with each other), and a CLI (e.g., a text-based user interface that allows a user 106 to interact with a computer program or operating system by typing commands into a command-line interpreter).

[0064] The chatbot interface 114 may include a translation component 206 configured to translate between natural language and formal query language utilized by the knowledge graph component 120 and knowledge graph 122. The knowledge graph component 120 may include a query engine 204. Queries are typically written or expressed by user 106 in natural language, and the query engine 204 can be used to translate the natural language query 128 from natural language into a formal query language, such as RDF query language or SPARQL. In some instances, the query engine 204 may utilize a translation engine (such as another LLM) to translate the query into query language. The query engine 204 can then use the knowledge graph 122 to evaluate the formal language query 128 and determine whether the knowledge graph 122 returns results. In some instances, the knowledge graph system 104 may also include a prepared knowledge graph 202 that can be used in conjunction with the knowledge graph 122 to answer query 128. The prepared knowledge graph 202 can be configured or constructed to answer questions specific to a particular subject domain, and the query engine 204 can determine which domains the prepared knowledge graph 202 should be used for query 128 that is semantically relevant. However, in an example where the query engine cannot use knowledge graph 122 and / or prepared knowledge graph 202 to identify the answer 134 to query 128, the knowledge graph component 120 passes query 128 to the prompting engineering component 208.

[0065] In some instances, a prepared knowledge graph 202 previously constructed or acquired by user 106 may be provided by user 106. Typically, knowledge graph 122 and the prepared knowledge graph 202 can be graph-like data structures with nodes, edges, attributes, and labels. Nodes in graphs 122 / 202 represent entities or concepts (e.g., person locations, objects, etc.), edges represent relationships between nodes, attributes provide additional metadata or information about the entities or relationships, and labels can be attached to nodes and edges to indicate the kind of entity or relationship being represented. Typically, individual words have their own nodes, and statements are edges in the graph connecting multiple nodes. For example, the expression “Fred was born in Germany” has the subject “Fred,” the predicate “place of birth,” and the object “Germany.” In this example, “place of birth” is an edge in graph 122 / 202, and the words “Fred” and “Germany” can be nodes connected by that edge “place of birth.” This is an example of a triple that can be represented in graph 122 / 202. In some instances, metadata can be appended to subjects, predicates, and / or objects, such as in the form of tags or attributes. Tags / attributes can be associated with each expression that includes related items. As described in more detail below, origin data indicating the source of the answer can be appended to nodes and / or edges included in the expression of the answer. In this way, when an answer is selected for a query, origin information indicating the source of that answer can be provided to the user 106 to indicate the source of the answer.

[0066] Query engine 204 can utilize various techniques to identify relevant information from knowledge graph 202 / 122, using one or more techniques to identify one or more nodes with relevant information as specified in formal language query 128. Query engine 204 can utilize graph traversal algorithms such as depth-first search (DFS) and breadth-first search (BFS) to traverse the nodes and edges of knowledge graph 202 / 122 and explore the graph structure to find relevant nodes and relationships. Query engine 204 can utilize pattern matching techniques to find subgraphs that match the structure specified in formal language query 128, which may involve identifying nodes and edges that match the query pattern. As explained herein, query engine 204 can utilize SPARQL (SPARQL protocol and RDF query language) as the query language, because SPARQL is a query language specifically designed for querying RDF (Resource Description Framework) data, which can be used to retrieve specific patterns or information from knowledge graph 122 / 202. Query engine 204 can utilize SPARQL queries to retrieve specific patterns or information from the graph. The query engine may additionally employ techniques such as indexing and optimizing the data structures in knowledge graphs 202 / 122, allowing for faster information lookup and retrieval, and inference and reasoning about knowledge graphs 122 / 202, which contain implicit information that can be inferred from explicit facts (e.g., inferring additional information from the graph based on logical rules). Query engine 204 may utilize semantic similarity and similarity measures, as well as formal language queries 128 for potentially finding nodes or edges similar to a given entity. Query engine 202 may use similarity measures, such as cosine similarity or the Jaccard index, to measure the similarity between nodes in knowledge graphs 122 / 202 to identify semantic similarity. Query engine 204 may utilize various statistical methods to estimate the probability of certain patterns appearing in knowledge graphs 122 / 202, and utilize heuristics to guide the search for relevant information. Heuristics are empirical rules that help query engine 204 prune the search space and focus on potentially relevant parts of the graph.

[0067] Query engine 204 can identify relevant nodes in knowledge graph 122 / 202 as answers to a given query. Query engine 204 (and / or chatbot system 110) can utilize retrieval-enhanced generation (RAG) to determine the answers provided by information stored in knowledge graph 122 / 202. RAG is an AI framework for retrieving facts from external knowledge bases to a base language model based on the most accurate and up-to-date information, and providing users with an understanding of the LLM generation process. RAG is a natural language processing (NLP) approach that combines retrieval-based systems and generative models. The purpose of the RAG approach is to enhance the capabilities of generative models by combining information retrieval techniques. Retrieval-based techniques include those used to find relevant information from knowledge graph 122 / 202. Retrieval techniques can include a variety of techniques, such as keyword matching, TF-IDF (term frequency-inverse document frequency), or more advanced methods, such as dense vector search (using models such as BERT sentence transformers) to retrieve relevant information. Generative techniques typically include generative models, such as language models capable of generating human-like text. The generative models used are powerful in generating coherent and context-relevant text, but they may not always have access to specific information or facts. Query engine 204 can integrate information retrieved from knowledge graphs 122 / 202 to provide context or enhance the generative model's response. This integration can be done in various ways, such as concatenating the retrieved information with input prompts, using it as a context window, or employing attention mechanisms to focus on specific parts of the retrieved content. Query engine 204 can then use the combined input (original prompts + retrieved information) passed to the generative model to generate output, and the model then generates a response based on this enhanced input. Query engine 204 and / or chatbot system 110 utilize RAG to provide answers to queries and strike a balance between the strength of the generative model (creative text generation) and the retrieval-based system (access to specific information). By combining these components, query engine 204 and / or chatbot system 110 aim to produce context-relevant and, in fact, accurate responses.

[0068] The query engine 204 and / or chatbot system 110 may utilize RAG to generate a response that remains in the same tone as the rest of its conversation with user 106, but is actually more likely to be based on it, and because it allows the language model to generate a natural vocal response based on multiple pieces of information derived from knowledge graphs 122 / 202.

[0069] Furthermore, the prompting engineering component 208 can perform various prompting engineering techniques. For example, the prompting engineering component 208 can provide the LLM 118 with a portion of the knowledge graph 122, such as the most relevant information for the formal language query 128. For example, the query engine 204 can determine which part of the knowledge graph 122 is semantically most relevant, or has an answer relevant to the formal language query 128. The prompting engineering component 208 can then provide the LLM 118 with the most relevant information along with the formal language query 128 in one or more prompts 132. Furthermore, instead of providing relevant information in the language or structure of the knowledge graph (e.g., semantic triples, RDF triples, etc.), the prompting engineering component 208 can generate a summary of the relevant information (possibly utilizing the LLM) to reduce the amount of data provided to the LLM 118.

[0070] Typically, hint 128 can be in a format understood by LLM 118. For example, hint 128 may include text such as “Answer the following query: ‘List the U.S. presidents born in the baby boomer generation and their dates of birth’ and use the following information from the knowledge graph to determine the answer [knowledge graph information].” LLM 118 may initially analyze hint 128, particularly the knowledge graph information (which may be in a formal language), to determine whether hint 128 fits the context window size of LLM 118, and perform the techniques described herein based on whether hint 128 is too large for the context window size (e.g., summarizing hint 128, providing only the most relevant information, etc.), or if hint 128 fits the context window size, LLM 118 may be used to analyze hint 128.

[0071] In addition to LLM 118, AI system 116 may also include a query language component 210, which is configured to translate back and forth between natural language and query language based on inputs and outputs in communication with prompting engineering component 208. For example, prompt 132 may include formal language query 128, natural language query 126, and / or combinations thereof, and query language component 210 may translate the query as it is analyzed to determine a response.

[0072] Service provider system 102 may optionally include various services and data structures, and may communicate with various remote or external services or data structures. For example, service provider system 102 may include internal data sources 218 (e.g., service documents) that can be used to improve knowledge graph 122. Internal data sources 218 may include data provided by user 106, documents generated by service provider, and / or other information.

[0073] To utilize the services provided by service provider system 102, users can register user accounts 220 with service provider system 102. For example, users can use user devices to interact with the Identity and Access Management (IAM) component, which allows users to create user accounts 220 using service provider system 102. Service provider system 102 can also provide and store user accounts 220, through which user 106 can interact with service provider system 102 and improve the knowledge graph 122 generated on their behalf. Typically, the IAM component enables user 106 to remotely manage their network infrastructure and view data provided by knowledge graph system 104. Typically, different user accounts 220 can assume different roles, sets, or licenses / credentials, allowing users to perform different actions and being restricted to performing certain actions. In some instances, the same organization may have multiple accounts with different network infrastructures.

[0074] Service provider system 102 may additionally include one or more graph services 222 that assist in the creation, development, and maintenance of knowledge graph 122. These graph services 222 can perform a variety of activities, ranging from data acquisition and cleaning to design patterns, linking entities, and establishing infrastructure for knowledge graph storage and retrieval. Graph services 222 can work with knowledge graph system 104 to create knowledge graph 122, potentially without requiring a dataset, as described herein.

[0075] Service provider system 102 may include various other services 224 that interact with or utilize the knowledge graph system 104. For example, other services 224 may provide chatbots that help users use their various functionalities. Other services 224 may utilize the knowledge graph system 104 to construct corresponding knowledge graphs 122 that improve the chatbots utilized by those other services 224.

[0076] As described herein, service provider system 102 may include one or more hardware processors (processors) or one or more devices configured to execute one or more stored instructions. The processor may include one or more cores. Furthermore, service provider system 102 may include one or more network interfaces configured to provide communication between service provider system 102 and other devices such as user equipment and / or other systems or devices within and / or remote from service provider system 102. Network interfaces may include devices configured to couple to personal local area networks (PANs), wired and wireless local area networks (LANs), wired and wireless wide area networks (WANs), etc.

[0077] Service provider system 102 may also include a computer-readable medium storing various executable components, such as software-based components, firmware-based components, etc. In addition to the various components discussed in the preceding figures, the computer-readable medium may also store components to implement the functionality described herein. Although not illustrated, the computer-readable medium may store one or more operating systems for controlling the operation of one or more devices including service provider system 102. According to one embodiment, the operating system includes a LINUX operating system. According to another embodiment, the operating system includes a WINDOWS SERVER operating system from MICROSOFT Corporation of Redmond, Washington. According to yet another embodiment, the operating system may include a UNIX operating system or a variant thereof. It should be understood that other operating systems may also be used.

[0078] Additionally, the service provider system 102 may include data storage or storage devices, which may include one or more repositories or other storage locations for persistently storing and managing collections of data such as databases, simple files, binary data, and / or any other data. The storage devices may include one or more storage locations that can be managed by one or more database management systems.

[0079] To utilize the services provided by Service Provider System 102, users can register accounts with Service Provider System 102. For example, users can use their user devices to interact with the Identity and Access Management (IAM) component, which allows users to create user accounts using Service Provider System 102. Typically, the IAM component enables users to remotely manage their network infrastructure and view data provided by Knowledge Graph System 104. Different user accounts may typically assume different roles, sets, or licenses / credentials, allowing users to perform different actions and being restricted to performing certain actions. In some instances, the same organization may have multiple accounts with different network infrastructures.

[0080] Computer-readable media can be used to store and retrieve information, such as program modules, data structures, or other data. Those skilled in the art will understand that a computer-readable storage medium is any available medium that provides non-transitory storage of data and can be accessed by the service provider system 102. In some examples, the operations performed by the service provider system 102 and / or any components included therein may be supported by one or more server devices. In other words, some or all of the operations performed by the service provider system 102 and / or any components included therein may be performed by one or more computer devices operating in a cloud-based configuration.

[0081] By way of example and not limitation, computer-readable storage media can include volatile and non-volatile, removable and non-removable media implemented in any method or technology. Computer-readable storage media include, but are not limited to, RAM, ROM, erasable programmable ROM (“EPROM”), electrically erasable programmable ROM (“EEPROM”), flash memory or other solid-state memory technologies, optical disc ROM (“CD-ROM”), digital universal disk (“DVD”), high-resolution DVD (“HD-DVD”), BLU-RAY or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information in a non-transitory manner.

[0082] Figure 3 Example timeline flow 300 is illustrated, according to which a query is processed through a pipeline, including the use of knowledge graph 122 and LLM 118 to determine the answer to the query.

[0083] Initially, user 106 can submit a natural language query 126 via chatbot UI 124 and user device 108. In this particular example, user 106 asks the presidents of the United States (US) which of them was born during the baby boomer generation and what their birth date is.

[0084] Natural language query 128 can be translated by chatbot system 110 using one or more translation LLMs and then provided to query engine 204. Translation LLMs (e.g., translation component 206) can translate natural language query 128 from natural language query 126 into query language query 128. Formal language query 128 can be used to query knowledge graph 122 to obtain answer 134.

[0085] In an instance where knowledge graph 122 does not return an answer 134 to formal language query 128, prompting engineering component 208 can then generate a prompt 132 for LLM 118. As shown, prompt 132 may include a request from LLM 118 to generate a true answer to query 128, may include natural language query 126, and may also include one or more triple patterns (e.g., some of which are lists of triples of variables). Prompting engineering component 208 can then generate prompt 132, which consists of this particular triple pattern and instructions on how to evaluate it. Prompting engineering component 208 may also include additional information in the prompt to help improve the accuracy of the LLM response, including the initial (plain language) question being answered and the inference type for each variable.

[0086] Then, LLM 118 can respond with a list of assignments 302 for each variable in the triplet pattern. In some examples, hint engineering component 208 may iteratively request LLM 118 to generate additional assignments 302 until it responds that no more assignments 302 are possible. Hint engineering component 208 may face context window limitations, and the length of hint 132 increases as the list of solutions already generated grows (although this list will be much smaller than the overall size of the generated knowledge graph 122). Therefore, hint engineering component 208 can replace parts of the solution list with (LLM-inferred) summaries. Chatbot component 112 receives these assignments 302 and generates a natural language response 136 for output to chatbot interface 114.

[0087] Figure 4A An example interface 404 of a chatbot system 110 is shown, which presents origin data 402 of the responses to queries and prompts the user 106 to report any errors in the natural language responses 136.

[0088] In some instances, 106 can request origin data 402 from chatbot system 110 via chatbot UI 124. Chatbot system 110 may be able to work with knowledge graph component 120 to provide origin data 402 for answer 134 to query 128. Origin data 402 may be stored in knowledge graph 122 and associated or mapped to the answer, such as by using labels or other metadata assigned to nodes and / or edges. Origin data 402 may indicate the origin or source of natural language answer 136, such as the URL of the data source, the name of the entity providing natural language answer 136, etc. Knowledge graph system 104 may store origin data 402 in knowledge graph 122 and map it to natural language answer 136. In this way, if user 106 requests the chatbot system 110 to provide indication of the source of the natural language response 136, the chatbot system 110 can work with the knowledge graph system 104 to obtain origin data 402 of response 136 (or multiple responses 136), and provide this information to user 106 via the chatbot UI 124. User 106 can then decide whether they trust the source of the natural language response 136 and subsequently verify the natural language response 136 themselves.

[0089] Furthermore, the chatbot UI 124 may include or display feedback mechanisms, such as a report highlighting error option 410, through which user 106 can provide feedback on the accuracy of the answers provided by the chatbot. In this example, user 106 may be able to highlight the portion of the natural language answer 136 as incorrect 406, or simply mark the entire answer as incorrect or confusing. Feedback data 412 may be provided to knowledge graph component 120 and used to improve knowledge graph 122.

[0090] Figure 4B Example system architecture diagram 400 illustrates where chatbot system 110 provides feedback data 412 for answer 134 and knowledge graph system 104 modifies knowledge graph 122 used to determine answer 134 based on the feedback data 412.

[0091] User 106 can choose the option 410 to report highlighted errors and can provide feedback data 412 to knowledge graph system 102. Knowledge graph component 120 and / or an administrator (administrator 418) associated with knowledge graph system 104 can determine whether answer 134 is accurate, and if so, modify or remove the answer. In some instances, remote answer validator 214 (e.g., external LLM, external subject matter expert, etc.) and / or remote data source 216 (e.g., web-based encyclopedia, user forum, etc.) can be used to determine whether answer 134 is accurate, and if so, modify or remove the answer. In this example, knowledge graph component 120 can determine that answer 134 is inaccurate or confusing, or receive input from other sources, and can perform graph modification 416 to change, modify, or remove answer 134 and improve knowledge graph 122.

[0092] Figure 5 An example system architecture diagram of example interface 500 is shown, through which engineering component 208 is prompted to provide relevant part 502 (or related part 502) of the knowledge graph to LLM 118 in order to help determine the answer to query 128.

[0093] To handle context window limitations, the hinting engineering component 208 can provide the LLM 118 with the most relevant information for query 128. For example, the hinting engineering component 208 can determine which part of the knowledge graph 122 is semantically most relevant, or has an answer relevant to the query. The hinting engineering component 208 can then provide the relevant part 502 along with query 128 to the LLM 118.

[0094] As shown in the figure, prompt 504 includes query 128, as well as some information or answer 134 already included in knowledge graph 112. Response 506 from LLM 118 may include some additional answer 134 to query 128. Prompt engineering component 208 may continue to iteratively prompt LLM 118 until LLM 118 has no more answers 134. As indicated by prompt 508, prompt engineering component 208 requests additional answers 134 again and provides a known answer 134 for query 128. LLM 118 may provide a response 510 instructing it to provide all information related to query 128.

[0095] Figure 6 An example interface 600 is shown where the prompting component 208 generates a summary prompt 608 within a context window suitable for LLM 118 through its summary initial prompt 604.

[0096] In the initial prompt 604, the prompt engineering component 208 may include a natural language query 128, and an assignment 302 that has been identified and stored in the knowledge graph 122. The response 606 provided by the LLM 118 indicates that the initial prompt 604 exceeds the context window size of the LLM 118. Therefore, the prompt engineering component 208 may generate a summary of the assignment 302 (possibly utilizing the LLM) and / or other data from the initial prompt to generate the prompt 608. The LLM 118 is then able to provide a response 610 that includes the answer 134 desired by the prompt engineering component 208.

[0097] In some instances, interfaces 500 and 600 may be chat interfaces through which users and / or programs can submit text (and other input) prompts. However, in some examples, interfaces 500 and 600 may be any type of interface through which software such as prompt engineering component 208 can communicate with LLM 118, such as CLI, API, or any other type of software instruction. Therefore, while some examples described and / or illustrated herein are shown as interfaces 500 and 600 being chat interfaces, these are merely illustrative, and interfaces 500 and 600 may include any type of software interface that prompt engineering component 208 (and / or user 106) can use to interface with the LLM.

[0098] Figure 7 , Figure 8 , Figure 9 , Figures 10A to 10B and Figure 11 Flowcharts illustrating example processes 700, 800, 900, 1000, and 1100 are provided, illustrating aspects of the functionality performed at least in part by the service provider system 102 as described in this disclosure. References herein Figure 7 , Figure 8, Figure 9 , Figures 10A to 10B and Figure 11 The described logical operations may (1) be implemented as a sequence of computer-implemented actions or program modules running on a computing system, and / or (2) be implemented as interconnected machine logic circuits or circuit modules within a computing system.

[0099] The implementation of the various components described herein depends on the choice of computing system performance and other requirements. Therefore, the logical operations described herein are referred to differently as operations, structural devices, actions, or modules. These operations, structural devices, actions, and modules can be implemented using software, firmware, dedicated digital logic, and any combination thereof. It should also be understood that executable... Figure 7 , Figure 8 , Figure 9 , Figures 10A to 10B and Figure 11 The diagram illustrates more or fewer operations than those described herein. These operations may also be performed in parallel or in a different order than those described herein. Some or all of these operations may also be performed by components other than those specifically identified. While the techniques described in this disclosure are referenced to specific components, in other examples, the techniques may be implemented with fewer components, more components, different components, or any configuration of components.

[0100] Figure 7 A flowchart illustrating an example process 700 in which a knowledge graph system 104 uses an LLM 118 to determine the answer to a query and add the answer to the query to the knowledge graph is shown.

[0101] At 702, the knowledge graph system 104 may obtain and / or generate a formal language query 128, which can be used to retrieve a formal language answer from the knowledge graph. The knowledge graph system 104 may receive a natural language query 126 via a chatbot system 110 and from a user 106, and in some instances, the knowledge graph system 104 may instruct one or more LLMs 118 to determine the query and the answer to the query to construct a knowledge graph 122. As another example, the knowledge graph system 104 may obtain a dataset including multiple queries related to one or more subject domain regions and iteratively prompt the LLMs 118 to determine a formal language answer 134 that can be used to generate the knowledge graph 122.

[0102] At 704, the knowledge graph system 104 can determine through query engine 204 that the formal language answer 134 to the formal language query 128 is not included in the knowledge graph 122. For example, the query engine 204 associated with the knowledge graph system 104 can query the knowledge graph 122 to identify the formal language answer 134 as the formal language query 128 and determine that the formal language answer 134 is not included therein.

[0103] At 706, the knowledge graph system 104 may prompt the Large Language Model (LLM) 118 to determine the answer to the formal language query 128, at least in part, based on the fact that the formal language answer 134 is not included in the knowledge graph 122. For example, the knowledge graph component 120 may generate a prompt 132 for the LLM 118 that includes the natural language query 126, the formal language query 128, and / or combinations thereof (as well as potentially relevant portions of the knowledge graph 502 and / or known answer summaries 602). In some instances, the knowledge graph system 104 may identify portions of the knowledge graph 122 that include information relevant to the formal language query 128 and generate a prompt 132 for the LLM 118 to determine the formal language answer 134 to the formal language query 128, wherein the prompt 132 includes that portion of the knowledge graph 122 and the formal language query 128.

[0104] At 708, the knowledge graph system 104 may receive the formal language answer 136 of the formal language query 128 as output from the LLM 118. At 710, the knowledge graph system 104 may add the formal language answer 134 to the knowledge graph 122 as a formal answer to the formal language query 128, and / or may use the output to modify the answer in the knowledge graph 122.

[0105] In some instances, process 700 may also include receiving feedback data 412 indicating that the answer is an incorrect answer to the query, and modifying the answer in the knowledge graph or removing the answer from the knowledge graph.

[0106] In some instances, process 700 may also include identifying a portion of the knowledge graph that includes information related to the query, and generating a hint for the LLM to determine an answer to the query, the hint including that portion of the knowledge graph and the query, wherein prompting the LLM to determine an answer includes providing a hint to the LLM.

[0107] In some instances, process 700 may also include prompting the LLM to determine a second answer to the query, receiving the second answer to the query as a second output from the LLM, generating a summary of known answers to the query using the first and second answers in natural language, wherein the summary of known answers includes a data size smaller than the context window of the LLM, and prompting the LLM to determine a third answer to the query, the prompt including providing the LLM with a summary of known answers.

[0108] Figure 8 A flowchart of an example process 800 is shown for a knowledge graph system 104 to iteratively prompt an LLM 118 to determine answers to multiple queries and to construct a knowledge graph 122 by adding the answers to the queries to the knowledge graph 122.

[0109] At 802, the knowledge graph system 104 may receive a request to construct the knowledge graph 122. In some instances, this request may come from a user account 220 registered with the graph service 222 that wishes to construct the knowledge graph 122 around one or more topic regions.

[0110] At 804, the knowledge graph system 104 may obtain multiple queries from the knowledge graph 122 to provide answers to. For example, the knowledge graph system 104 may obtain multiple queries from a frequently asked questions (FAQ) database, from a customer help forum, etc. However, in some instances, the knowledge graph system 104 may simply require the LLM 118 to build the knowledge graph 122 around one or more topic areas by continuing to iterate around them.

[0111] At point 806, the knowledge graph system 104 iteratively prompts the LLM 118 to answer multiple queries. For example, the prompting engineering component 208 can continue to prompt the LLM 118 for answers to queries.

[0112] At 808, the knowledge graph system 104 can receive answers to multiple queries as output from the LLM. For example, the LLM 118 can continue to output answers to queries presented by the prompting engineering component 208.

[0113] At point 810, the knowledge graph system 104 can add answers to multiple queries to the knowledge graph 122. In this way, the LLM 118 can construct or generate the knowledge graph 122 through prompting engineering.

[0114] Figure 9 A flowchart illustrating an example process 900 in which chatbot system 110 receives a query in natural language, translates it into query language, and determines that knowledge graph 122 does not have an answer to the query. Chatbot system 110 then uses LLM 118 to determine the answer, adds it to knowledge graph 112, and provides the answer to user 106.

[0115] At 902, the chatbot system 110 may receive a query from the user 106 for which the chatbot system 110 will provide an answer. In such an example, the query may be expressed in natural language presented by the user 106.

[0116] At 904, the chatbot system 110 can translate queries from natural language into a query language associated with a knowledge graph. For example, the chatbot system 110 may include a translation component (e.g., LLM or other AI model) that translates natural language text into formal / query language text.

[0117] At 906, the query engine can query the knowledge graph to identify the answer to the query. For example, the query engine 204 associated with the knowledge graph system 104 can query the knowledge graph 122 to identify the formal language answer 134 as the formal language query 128 and determine that the formal language answer 134 is not included therein.

[0118] In some instances, the query engine may determine that the knowledge graph does not include an answer to the query, and the knowledge graph component 120 may prompt the LLM to determine an answer to the query. For example, the knowledge graph component 120 may generate a hint 132 for the LLM 118 that includes the natural language query 126, the formal language query 128, and / or combinations thereof (as well as potentially relevant portions of the knowledge graph 502 and / or known answer summaries 602). In some instances, the knowledge graph system 104 may identify portions of the knowledge graph 122 that include information related to the formal language query 128 and generate a hint 132 for the LLM 118 to determine a formal language answer 134 to the formal language query 128, wherein the hint 132 includes that portion of the knowledge graph 122 and the formal language query 128.

[0119] At 908, knowledge graph component 120 can provide the LLM with information associated with the one or more nodes to generate a response to the query. Knowledge graph component 120 can receive answers as output from the LLM. Knowledge graph component 120 can add the answers to the query to the knowledge graph. For example, knowledge graph system 104 can add a formal language answer 134 to knowledge graph 122 as a formal answer to formal language query 128, and / or use the output to modify the answers in knowledge graph 122.

[0120] The chatbot system 110 can generate a natural language response including the answer, and at 910, the chatbot system 110 can provide the natural language response to the user 106 via the chatbot interface 114.

[0121] Figure 10A and Figure 10B A flowchart of an example process 1000 is illustrated, which uses the answer from an LLM to build a knowledge graph, receives a query from a user, and then determines whether the knowledge graph or the LLM can answer the query.

[0122] Knowledge graph component 120 iteratively prompts the large language model (LLM) to determine answers to multiple queries. For example, knowledge graph system 104 may request LLM 118 to construct knowledge graph 122 around one or more topic regions by continuing to iterate around them. At 1004, knowledge graph component 120 may receive answers to multiple queries as output from the LLM.

[0123] Knowledge graph component 120 can add answers to the knowledge graph used by the chatbot to respond to users. For example, knowledge graph system 104 can add formal language answer 134 as a formal answer to formal language query 128 to knowledge graph 122, and / or modify the answers in knowledge graph 122.

[0124] At point 1002, the chatbot system 110 may receive a query from user 106 for which the chatbot system 110 will provide an answer. In such an example, the query may be expressed in natural language presented by user 106.

[0125] At 1004, the chatbot system 110 can translate queries from natural language into query language associated with a knowledge graph. For example, the chatbot system 110 may include a translation component (e.g., LLM or other AI model) that translates natural language text into formal / query language text.

[0126] At 1006, the query engine can query the knowledge graph to identify one or more nodes in knowledge graph 122 that are related to query 128. For example, query engine 204 associated with knowledge graph system 104 can query knowledge graph 122 to identify formal language answer 134 as formal language query 128 and determine that formal language answer 134 is not included therein.

[0127] In some instances, the query engine may determine that the knowledge graph does not include an answer to the query, and the knowledge graph component 120 may prompt the LLM to determine an answer to the query. For example, the knowledge graph component 120 may generate a hint 132 for the LLM 118 that includes the natural language query 126, the formal language query 128, and / or combinations thereof (as well as potentially relevant portions of the knowledge graph 502 and / or known answer summaries 602). In some instances, the knowledge graph system 104 may identify portions of the knowledge graph 122 that include information related to the formal language query 128 and generate a hint 132 for the LLM 118 to determine a formal language answer 134 to the formal language query 128, wherein the hint 132 includes that portion of the knowledge graph 122 and the formal language query 128.

[0128] At 1008, knowledge graph component 120 may provide the LLM with information associated with the one or more nodes to generate a response to the query. Knowledge graph component 120 may receive the answer as output from the LLM. Knowledge graph component 120 may add the answer to the query to the knowledge graph. For example, knowledge graph system 104 may add formal language answer 134 to knowledge graph 122 as a formal answer to formal language query 128, and / or may use the output to modify the answer in knowledge graph 122. At 1010, language model may receive information associated with one or more nodes. For example, prompting engineering component 208 submits prompt 128 including information expressed in one or more nodes.

[0129] At 1012, the chatbot system 110 can generate a natural language response including the answer, and at 1014, the chatbot system 110 can provide a natural language response to the user 106 via the chatbot interface 114.

[0130] At 1010, the language model may receive information associated with one or more nodes (e.g., hints 132 including information expressed in one or more nodes). At 1012, the language model may generate at least a portion of the answer to the query.

[0131] At 1014, the chatbot system 110 can provide answers to a user via the chatbot. This may include providing the user with answers from the knowledge graph, or providing the language model with answers from the knowledge graph to generate output to be provided to the user. Retrieval-enhanced generation (RAG) retrieves data from outside the language model and enhances the provided prompts by adding relevant retrieved data to the context. RAG can help reduce model illusions by guiding the output to be similar to or based on the retrieved information. Therefore, information from one or more nodes in the knowledge graph can be retrieved and added to the context window of the language model.

[0132] In some instances, in response to determining that an answer is not included in the knowledge graph, the knowledge graph component 120 may prompt the language model to determine the answer to the query, receive the answer to the query as output from the language model, add the answer to the knowledge graph, and provide the answer to the user via a chatbot. Additionally, security measures may be implemented in the generated knowledge graph, such as preventing the representation of certain concepts in the knowledge graph or ensuring that the knowledge graph maintains certain desired properties. As each new triple is added, the formal specification of the knowledge graph can be evaluated, and triples that would cause the knowledge graph to violate those desired properties can be rejected.

[0133] At point 1016, the output generated using nodes in the knowledge graph can also be provided with origin information. The information in the node is associated with its origin (e.g., links, citations, or other identifying information about the original source, such as author, title, etc., or confirmation that it was AI-generated). When an answer is generated based on these nodes, user 106 may also receive origin information, for example, as a footnote embedded in the response or as a separate origin report linked to the answer. Alternatively, origin information may be provided only upon request if the user explicitly requests it, or if the answer is problematic and the user wishes to fundamentally address errors in the response by checking the knowledge graph for errors. Another alternative is to provide origin information in systems where interpretability is critical (e.g., domain-specific medical AI assistants where doctors want to understand why the assistant can recommend specific diagnoses or treatments before completing a patient's plan; domain-specific legal AI assistants where lawyers need to physically examine their output against laws and cases that influence it).

[0134] Figure 11 This document illustrates a flowchart of an example process where a chatbot receives a query from a user's device, uses an LLM to determine the response to the query, and adds the response to the query to a knowledge graph for future use.

[0135] At 1102, the chatbot system 110 can provide the user device with access to a chatbot interface configured to receive queries for the chatbot. At 1104, the chatbot system 110 can receive queries from the user device via the chatbot interface. For example, the knowledge graph system 104 can receive a natural language query 126 from the user 106 via the chatbot system 110.

[0136] At 1106, the chatbot system 110 can determine that the answer to the query is not included in the knowledge graph used by the chatbot. For example, the query engine 204 associated with the knowledge graph system 104 can query the knowledge graph 122 to identify the formal language answer 134 as the formal language query 128 and determine that the formal language answer 134 is not included therein.

[0137] At 1108, the chatbot system 110 may determine the answer to a query based at least in part on a language model associated with the chatbot, provided that the answer is not included in the knowledge graph. For example, the knowledge graph component 120 may generate a hint 132 for LLM 118, which includes natural language query 126, formal language query 128, and / or combinations thereof (as well as potentially relevant portions of knowledge graph 502 and / or a known answer summary 602).

[0138] At 1110, the chatbot system 110 can receive the response to the query as output from the language model. At 1112, the chatbot system 110 can add the response to the query to the knowledge graph. For example, the knowledge graph system 104 can add the formal language response 134 to the knowledge graph 122 as a formal response to the formal language query 128, and / or can use the output to modify the response in the knowledge graph 122. At 1114, the chatbot system 110 can provide the response to the user device via a chatbot interface.

[0139] Figure 12 This is a system and network diagram 1200 illustrating an exemplary operating environment of a data center 1204 in one or more zones 1206 of a service provider system 102, including aspects that can be configured to implement the functionalities described herein. The service provider system 102 may provide computing resources, such as VM instances and storage, on a permanent or on-demand basis. Among other types of functionality, the computing resources provided by the service provider system 102 can be used to implement the various services described above. Also as discussed above, the computing resources provided by the service provider system 102 may include various types of computing resources, such as data processing resources, such as VM instances, data storage resources, networking resources, data communication resources, network services, etc.

[0140] Each type of computing resource provided by service provider system 102 can be general-purpose or available in many specific configurations. For example, in several different configurations, data processing resources can be provided as physical computers or VM instances. VM instances can be configured to run applications, including web servers, application servers, media servers, database servers, game applications, some or all of the aforementioned network services and / or other types of programs. Data storage resources can include file storage devices, block storage devices, etc. Service provider system 102 can also be configured to provide other types of computing resources not specifically mentioned herein.

[0141] In one implementation, the computing resources provided by the service provider system 102 may be enabled by one or more data centers 1204A to 1204N (which may be referred to herein solely as "data center 1204" or in the plural as "data center 1204"). Data center 1204 is a facility for housing and operating computer systems and related components. Data center 1204 typically includes redundant and backup power, communication, cooling, and security systems. Data center 1204 may also be located in geographically distinct locations or districts 1206. References will follow below. Figure 13 An exemplary implementation of a data center 1204 that can be used to implement the techniques disclosed herein is described.

[0142] User 106 may use user equipment 108 to access service provider system 102 and computing resources provided by service provider system 102 via any wired and / or wireless network 130, which may be a wide area communication network (“WAN”), such as the Internet, an intranet, or an Internet service provider (“ISP”) network, or a combination of such networks. For example, but not limited to, user equipment 108 operated by user 106 of service provider system 102 may be used to access service provider system 102 via network 130. It should be understood that local area network (“LAN”), the Internet, or any other network topology known in the art for connecting data center 1204 to remote clients and other users may be utilized. It should also be understood that combinations of such networks may also be utilized.

[0143] Figure 13 Figure 1300 illustrates a configuration of a data center 1204 used to implement various aspects of the techniques disclosed herein. Figure 13 The example data center 1204 shown may include a plurality of server computers 1302A to 1302F (which may be collectively referred to herein as "server computers 1302" in the singular or "a plurality of server computers 1302" in the plural) for providing computing resources 1304A to 1304E. In some examples, resources 1304 and / or server computers 1302 may include, be included in or correspond to the computing devices described herein.

[0144] Server computer 1302 may be a properly configured standard tower, rack, or blade server computer used to provide the computing resources described herein (in... Figure 13 Examples are shown as computing resources 1304A to 1304E. As mentioned above, the computing resources provided by the service provider system 102 can be data processing resources, such as VM instances or hardware computing systems, database clusters, computing clusters, storage clusters, data storage resources, database resources, networking resources, etc. Some servers 1302 may also be configured to run a resource manager 1302 capable of instantiating and / or managing computing resources. For example, in the case of VM instances, the resource manager 1302 may be a virtual machine monitor or another type of program configured to enable the execution of multiple VM instances on a single server computer 1302. The server computers 1302 in data center 1204 may also be configured to provide network services and other types of services.

[0145] exist Figure 13The example data center 1204 shown, with appropriate LAN 1308, is also used to interconnect server computers 1302A-1302F. It should be understood that the configurations and network topologies described herein have been greatly simplified, and many other computing systems, software components, networks, and network devices can be used to interconnect the various computing systems disclosed herein and provide the aforementioned functionality. Appropriate load balancing devices or other types of network infrastructure components can also be used to balance the load between each data center in data centers 1304A through 1304N, between each server computer in server computers 1302A through 1302F in each data center 1204, and potentially within each server computer in server computer 1302. It should be understood that references... Figure 13 The configuration of data center 1204 described is merely illustrative and may be implemented using other methods.

[0146] Figure 14 An example computer architecture is shown for a computer 1400 capable of executing program components for achieving the above-described functionality. Figure 14 The computer architectures shown illustrate conventional server computers, workstations, desktop computers, laptop computers, tablet computers, network appliances, e-readers, smartphones, or other computing devices, and can be used to execute any of the software components presented herein.

[0147] Computer 1400 includes a substrate 1402 or “motherboard,” which is a printed circuit board to which multiple components or devices can be connected via a system bus or other electrical communication paths. In one exemplary configuration, one or more central processing units (“CPU”) 1404 operate in conjunction with chipset 1406. CPU 1404 may be a standard programmable processor that performs the arithmetic and logic operations required to perform the operation of computer 1400.

[0148] The CPU 1404 performs operations by manipulating switching elements that distinguish and change discrete physical states, transitioning from one discrete physical state to another. Switching elements typically include electronic circuitry, such as flip-flops, that maintains one of two binary states, and electronic circuitry, such as logic gates, that provides an output state based on a logical combination of the states of one or more other switching elements. These basic switching elements can be combined to create more complex logic circuits, including registers, adder-subtractor units, arithmetic logic units, floating-point units, and more.

[0149] Chipset 1406 provides an interface between CPU 1404 and the remaining components and devices on substrate 1402. Chipset 1406 provides an interface to RAM 1408, which serves as main memory in computer 1400. Chipset 1406 may also provide an interface to computer-readable storage media such as read-only memory (“ROM”) 1410 or non-volatile RAM (“NVRAM”) for storing basic routines that help start computer 1400 and transfer information between various components and devices. ROM 1410 or NVRAM may also store other software components required for the operation of computer 1400 according to the configuration described herein.

[0150] Computer 1400 can operate in a networked environment using logical connections to remote computing devices and computer systems via a network such as network 130. Chipset 1406 may include functionality for providing network connectivity via a network interface controller (NIC) 1412, such as a Gigabit Ethernet adapter. NIC 1412 is capable of connecting computer 1400 to other computing devices via network 1108 or network 130. It should be understood that multiple NICs 1412 may be present in computer 1400 to connect the computer to other types of networks and remote computer systems.

[0151] Computer 1400 may include a storage device 1414 (e.g., a disk drive) that provides non-volatile storage for the computer. Storage device 1414 may consist of one or more physical storage cells. Storage device 1414 can store information by altering the magnetic properties of a specific location within a disk drive cell, the reflection or refraction properties of a specific location in an optical storage cell, or the electrical properties of a specific capacitor, transistor, or other discrete component in a solid-state storage cell. Other variations of the physical medium are possible without departing from the scope and spirit of this specification, wherein the foregoing examples are provided for ease of description only. Computer 1400 can also read information from storage device 1414 by detecting the physical state or characteristics of one or more specific locations within the physical storage cells.

[0152] In addition to the aforementioned storage device 1414, computer 1400 may also access other computer-readable storage media to store and retrieve information, such as program modules, data structures, or other data. Those skilled in the art will understand that a computer-readable storage medium is any available medium that provides non-transitory storage of data and can be accessed by computer 1400. In some examples, the operations performed by service provider system 102 and / or any components included therein may be supported by one or more devices similar to computer 1400. In other words, some or all of the operations performed by service provider system 102 and / or any components included therein may be performed by one or more computer devices 1400 operating in a network-based arrangement.

[0153] By way of example and not limitation, computer-readable storage media can include volatile and non-volatile, removable and non-removable media implemented in any method or technology. Computer-readable storage media include, but are not limited to, RAM, ROM, erasable programmable ROM (“EPROM”), electrically erasable programmable ROM (“EEPROM”), flash memory or other solid-state memory technologies, optical disc ROM (“CD-ROM”), digital universal disk (“DVD”), high-resolution DVD (“HD-DVD”), BLU-RAY or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information in a non-transitory manner.

[0154] Storage device 1414 may store an operating system used to control the operation of computer 1400. According to one embodiment, the operating system includes a LINUX operating system. According to another embodiment, the operating system includes a WINDOWS SERVER operating system from MICROSOFT Corporation of Redmond, Washington. According to yet another embodiment, the operating system may include a UNIX operating system or a variant thereof. It should be understood that other operating systems may also be used. Storage device 1414 may store other systems or applications and data used by computer 1400.

[0155] In one embodiment, storage device 1414, RAM 1408, ROM 1410, and / or other computer-readable storage media may be encoded with computer-executable instructions that, when loaded into computer 1400, transform the computer from a general-purpose computing system into a special-purpose computer capable of implementing the embodiments described herein. As described above, these computer-executable instructions transform computer 1400 by specifying how CPU 1404 transitions between states. According to one embodiment, computer 1400 has access to a computer-readable storage medium storing computer-executable instructions that, when executed by computer 1400, perform the various techniques described above. Computer 1400 may also include a computer-readable storage medium thereon storing instructions for performing any other computer implementation of the operations described herein.

[0156] Computer 1400 may also include one or more input / output controllers 1416 for receiving and processing input from a variety of input devices such as a keyboard, mouse, touchpad, touchscreen, electronic stylus, or other types of input devices. Similarly, input / output controllers 1416 may provide output to a display such as a computer monitor, flat panel display, digital projector, printer, or other types of output devices. It should be understood that computer 1400 may not necessarily... Figure 14 All components shown are included, and may include Figure 14 Other components not explicitly shown in the document, or those that can be used with Figure 14 The architecture shown is completely different.

[0157] While the foregoing invention has been described with respect to specific embodiments, it should be understood that the scope of the invention is not limited to these specific embodiments. Since other modifications and variations have been made to suit specific operational requirements and environments, which will be apparent to those skilled in the art, the invention is not to be considered limited to the examples chosen for purposes of disclosure, and covers all changes and modifications that do not constitute a departure from the true spirit and scope of the invention.

[0158] Although this application describes embodiments with specific structural features and / or methodological actions, it should be understood that the claims are not necessarily limited to the specific features or actions described. Rather, the specific features and actions are merely illustrative, and some embodiments fall within the scope of the claims of this application.

[0159] The embodiments of this disclosure may be described in light of the following provisions: 1. A system comprising: one or more processors; and one or more computer-readable media storing computer-executable instructions, which, when executed by the one or more processors, cause the one or more processors to: provide a first virtual computing resource on computing resources associated with a service provider network managed by a service provider, wherein the first virtual computing resource is a first virtual computing resource type and the first virtual computing resource type is allocated a first amount of the computing resources for utilization; deploy workloads supported by the first virtual computing resource on behalf of user accounts registered with the service provider network, wherein the workloads are associated with resource utilization characteristics indicative of the utilization of the computing resources by the workloads; and identify a second virtual computing resource type, the second virtual computing resource... The type has been made available to support workloads on behalf of user accounts registered with the service provider network, wherein a second virtual computing resource type is allocated a second amount of computing resources for utilization; the resource utilization characteristics determine that the second virtual computing resource type is more optimized than the first virtual computing resource type to support the workload; the user account is provided with recommendation data, including recommendations, to migrate the workload from being supported by the first virtual computing resource type to being supported by the second virtual computing resource type; input data indicating a request from the user account to migrate the workload from being supported by the first virtual computing resource type to being supported by the second virtual computing resource type; and the user account migrates the workload supported by the second virtual computing resource type on behalf of the user account.

[0160] 2: A system as described in Clause 1, the system comprising additional computer-executable instructions, which, when executed by the one or more processors, cause the one or more processors to: provide a simulated virtual computing resource on the computing resource, wherein the simulated virtual computing resource is a second virtual computing resource type, the second virtual computing resource type being allocated a second amount of computing resources for utilization; cause a simulation program to execute on the simulated virtual computing resource, such that the simulation program utilizes the computing resources of the simulated virtual computing resource according to the resource utilization characteristics associated with the workload; and receive from the simulation program output data indicating the performance of the simulation program executing on the simulated virtual computing resource, wherein determining that the second virtual computing resource type is more optimized than the first virtual computing resource type to support the workload is at least partially based on the output data.

[0161] 3: A system as described in any one of Clauses 1 or 2, wherein the recommendation data further comprises: suitability data, the suitability data indicating a metric of the suitability of the second virtual computing resource type for supporting the workload; and text data, the text data comprising a textual explanation of the suitability of the second virtual computing resource type for supporting the workload.

[0162] 4: A system as described in any one of Clauses 1, 2, or 3, wherein: the second virtual computing resource type is optimized to support workloads utilizing a specific computing resource type; the specific computing resource type includes at least one of a central processing unit (CPU) resource type, a memory resource type, a storage resource type, or a network availability resource type; and the resource utilization characteristics indicate that the workload utilizes computing resources of the specific computing resource type.

[0163] 5. A computer-implemented method comprising: hosting a workload on behalf of a user account using at least partially a first virtual computing resource provided on a first computing resource in a service provider network, wherein the first virtual computing resource is a first virtual computing resource type allocated to the first computing resource for utilization; determining that the workload changes from utilizing a first amount of the first computing resource to utilizing a second amount of the first computing resource; identifying a second virtual computing resource type, more optimized than the first virtual computing resource type for hosting the workload, based at least partially on the second amount of the first computing resource, wherein the second virtual computing resource type is allocated to the second computing resource for utilization; and providing the user account with recommendation data, the recommendation data including a recommendation to migrate the workload from hosting by the first virtual computing resource to hosting by the second virtual computing resource, wherein the second virtual computing resource is the second virtual computing resource type.

[0164] 6: The computer-implemented method as described in Clause 5 further includes: receiving from the user account an indication that the workload has been modified; and at least in part in response to receiving the indication that the workload has been modified, collecting a resource utilization metric of the first computing resource indicating the second amount.

[0165] 7: A computer-implemented method as described in any one of clauses 5 or 6, further comprising: determining that the difference between the first amount of computing resources and the second amount of computing resources is greater than a threshold difference, wherein identifying the second virtual computing resource type is performed at least in part based on the difference being greater than the threshold difference.

[0166] 8: A computer-implemented method as described in any one of clauses 5, 6 or 7, further comprising: receiving input data instructing the user account to migrate the workload from being hosted by the first virtual computing resource to being hosted by the second virtual computing resource; and migrating the workload to be hosted by the second virtual computing resource on behalf of the user account.

[0167] 9. A computer-implemented method as described in any one of clauses 5, 6, 7, or 8, further comprising: providing a test virtual computing resource on a third computing resource associated with the service provider network, the test virtual computing resource being the second virtual computing resource type; deploying a test workload to be hosted by the test virtual computing resource, wherein the test workload utilizes the second amount of the third computing resource; receiving a health metric indicating the performance of the test workload while it is hosted on the test virtual computing resource; and migrating the workload to be hosted by the second virtual computing resource on behalf of the user account, at least in part based on the health metric.

[0168] 10: A computer-implemented method as described in any one of clauses 5, 6, 7, 8 or 9, further comprising: collecting a first utilization value of the first computing resource indicating the first amount from the first virtual computing resource at a first time; determining that a period of time has elapsed since the first time, wherein the period of time is associated with the frequency at which the user account modifies the workload; and collecting a second utilization value of the first computing resource indicating the second amount from the first virtual computing resource at a second time.

[0169] 11: A computer-implemented method as described in any one of clauses 5, 6, 7, 8, 9, or 10, further comprising: receiving from the user account a request to provide recommendations regarding the migration of the workload; and at least in part in response to receiving the request, collecting a resource utilization metric of the first computing resource indicating the second amount.

[0170] 12: A computer-implemented method as described in any one of clauses 5, 6, 7, 8, 9, 10, or 11, further comprising providing the user account with at least one of the following: suitability data indicating a measure of suitability of the second virtual computing resource type for hosting the workload; or text data including a textual explanation of the suitability of the second virtual computing resource type for hosting the workload.

[0171] 13: A computer-implemented method as described in any one of clauses 5, 6, 7, 8, 9, 10, 11, or 12, wherein: allocating a first amount of central processing unit (CPU) resources, including the first amount in the first computing resource, to a first virtual computing resource; allocating a second amount of CPU resources, including the second amount in the second computing resource, to a second virtual computing resource, the second amount of CPU resources being greater than the first amount of CPU resources; and determining that the workload changes from utilizing the first amount of the first computing resource to utilizing the second amount of the first computing resource, including determining that the workload is utilizing additional CPU resources.

[0172] 14: A system comprising: one or more processors; and one or more computer-readable media storing computer-executable instructions, which, when executed by the one or more processors, cause the one or more processors to: at least partially use a first virtual computing resource provided on a first computing resource of a service provider network to host workloads on behalf of user accounts, wherein the first virtual computing resource is a first virtual computing resource type allocated to the first computing resource for utilization; identify a second virtual computing resource type that has been made available for hosting workloads on behalf of user accounts associated with the service provider network, wherein the second virtual computing resource type is allocated a second computing resource for utilization; receive resource utilization characteristics indicating the utilization of the first computing resource by the workload; determine, based on the resource utilization characteristics, that the second virtual computing resource type is more optimized than the first virtual computing resource type for hosting the workload; and provide recommendation data to the user account, the recommendation data including a recommendation to migrate the workload from hosting by the first virtual computing resource to hosting by the second virtual computing resource, wherein the second virtual computing resource is the second virtual computing resource type.

[0173] 15: A system as described in Clause 14, the system comprising additional computer-executable instructions, which, when executed by the one or more processors, cause the one or more processors to: provide a simulated virtual computing resource on a third computing resource, wherein the simulated virtual computing resource is the second virtual computing resource type; cause a simulation program to execute on the simulated virtual computing resource, such that the simulation program utilizes the third computing resource of the simulated virtual computing resource according to the resource utilization characteristics associated with the workload; and receive from the simulation program output data indicating the performance of the simulation program executing on the simulated virtual computing resource, wherein determining that the second virtual computing resource type is more optimized than the first virtual computing resource type to host the workload is at least partially based on the output data.

[0174] 16: A system as described in any one of Clauses 14 or 15, the system comprising additional computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to: receive input data instructing the user account to migrate the workload from being hosted by the first virtual computing resource to being hosted by the second virtual computing resource; and, on behalf of the user account, migrate the workload to be supported by the second virtual computing resource.

[0175] 17: A system as described in any one of Clauses 14, 15, or 16, wherein the recommendation data further comprises at least one of: suitability data indicating a metric of the suitability of the second virtual computing resource type for hosting the workload; or text data comprising a textual explanation of the suitability of the second virtual computing resource type for hosting the workload.

[0176] 18: A system as described in any one of Clauses 14, 15, 16, or 17 further includes computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to: provide a test virtual computing resource on a third computing resource associated with the service provider network, the test virtual computing resource being of the second virtual computing resource type; deploy test workloads to be hosted by the test virtual computing resource, wherein the test workloads utilize the second amount of the third computing resource; receive a health metric indicating the performance of the test workloads while they are hosted on the test virtual computing resource; and migrate the workloads to be hosted by the second virtual computing resource on behalf of the user account, based at least in part on the health metric.

[0177] 19. A system as described in any one of Clauses 14, 15, 16, 17 or 18, wherein: the second virtual computing resource type is optimized to support workloads utilizing a specific computing resource type; the specific computing resource type includes at least one of a central processing unit (CPU) resource type, a memory resource type, a storage resource type or a network availability resource type; and the resource utilization characteristics indicate that the workload utilizes computing resources of the specific computing resource type.

[0178] 20: A system as described in any one of clauses 14, 15, 16, 17, 18, or 19 further includes computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to: determine a first optimized value associated with a first virtual computing resource type hosting the workload; determine a second optimized value associated with a second virtual computing resource type hosting the workload; determine that the second optimized value is greater than the first optimized value by a threshold amount; and determine to provide the recommended data to the user account based at least in part on the second optimized value being greater than the first optimized value by the threshold amount.

[0179] 21: A computer-implemented method for a chatbot to provide answers to queries using a knowledge graph constructed using a Large Language Model (LLM), the computer-implemented method comprising: receiving a query expressed in natural language at the chatbot and from a user; translating the query from the natural language into a query language associated with the knowledge graph; querying the knowledge graph by a query engine to identify one or more nodes in the knowledge graph associated with the query; providing information associated with the one or more nodes to the LLM to generate a response to the query; and providing the response in natural language to the user via the chatbot.

[0180] 22: The computer-implemented method as described in Clause 21 further includes: receiving feedback data from the user via the chatbot indicating that the answer is an incorrect answer to the query; and modifying the answer in the knowledge graph or removing at least one of the answers from the knowledge graph.

[0181] 23: A computer-implemented method as described in any one of clauses 21 or 22, further comprising: receiving a request from the user via the chatbot to provide an indication of the source of the answer provided by the chatbot; identifying origin data stored in the knowledge graph indicating that the source of the answer is the LLM; and providing the user via the chatbot with the indication that the source of the answer is the LLM.

[0182] 24: A computer-implemented method as described in any one of clauses 21, 22, or 23, further comprising: receiving via the chatbot feedback data indicating that the user is dissatisfied with the answer provided for the query; prompting the LLM to determine a second answer to the query; receiving the second answer to the query as a second output from the LLM; adding the second answer to the query to the knowledge graph; and providing the user with a second response including the second answer to the query.

[0183] 25: A system comprising: a chatbot component configured to perform a first operation, the first operation including: receiving a query to be answered by the chatbot component from a user via a chatbot interface; and providing the answer to the query to the user via the chatbot interface; a knowledge graph component configured to perform a second operation, the second operation including: converting the query from natural language into a query language associated with the knowledge graph; using the query in the query language to identify one or more nodes of the knowledge graph associated with the query; providing information associated with the one or more nodes to a language model to generate a response to the query; and a language model configured to perform a third operation, the third operation including: receiving the information associated with the one or more nodes; and using the information to generate at least a portion of the answer to the query.

[0184] 26: The system as described in Clause 25, wherein the knowledge graph is associated with a user account and is a first knowledge graph associated with a first topic domain, the second operation performed by the knowledge graph component further includes: determining that the user account is associated with a second knowledge graph, the second knowledge graph being associated with a second topic domain; determining that the query is associated with the first topic domain; and determining, at least in part, to suggest the first knowledge graph rather than the second knowledge graph for the answer based on the association of the query with the first topic domain.

[0185] 27: A system as described in any one of Clauses 25 or 26, wherein: the first operation performed by the chatbot component further includes: receiving feedback data from the user via the chatbot interface indicating that the answer is an incorrect answer to the query; and the second operation performed by the knowledge graph component further includes: identifying a portion of the knowledge graph used to determine the answer to the query; and modifying or removing at least one of the portions of the knowledge graph.

[0186] 28: A system as described in any one of clauses 25, 26, or 27, wherein: the first operation performed by the chatbot component further includes: receiving a request from the user via the chatbot interface to provide an indication of the source of the answer provided by the chatbot; the second operation performed by the knowledge graph component further includes: identifying origin data stored in the knowledge graph indicating that the source of the answer is the language model; and providing the user via the chatbot with the indication that the source of the answer is the language model.

[0187] 29: A system as described in any one of clauses 25, 26, 27, or 28, wherein: the first operation performed by the chatbot component further includes receiving feedback data via the chatbot interface indicating that the user is dissatisfied with the answer provided for the query; the second operation performed by the knowledge graph component further includes: prompting the language model to determine a second answer to the query; receiving the second answer to the query as a second output from the language model; adding the second answer to the query to the knowledge graph; and providing the user with a response including the second answer to the query.

[0188] 30: A system as described in any one of clauses 25, 26, 27, 28, or 29, wherein: the first operation performed by the chatbot component further includes receiving, via the chatbot interface, a request for an explanation of why the answer to the query is provided; the second operation performed by the knowledge graph component further includes: identifying a portion of the knowledge graph for determining the answer to the query; and using information in the portion of the knowledge graph to generate the explanation of why the answer to the query is provided; and the first operation performed by the chatbot component further includes providing the explanation to the user via the chatbot interface.

[0189] 31: A system as described in any one of clauses 25, 26, 27, 28, 29, or 30, wherein the answer is a first answer and the query is a first query, and wherein: the second operation performed by the knowledge graph component further includes: prompting the language model to determine a second answer to a second query; receiving the second answer to the query as a second output from the language model; determining that the second answer is associated with a concept that is not allowed to be represented in the knowledge graph; and avoiding adding the second answer to the knowledge graph.

[0190] 32: In a system as described in any one of clauses 25, 26, 27, 28, 29, 30, or 31, the second operation performed by the knowledge graph component further includes: identifying a portion of the knowledge graph that includes information related to the query; and generating a prompt for the language model to determine the answer to the query, the prompt including the portion of the knowledge graph and the query, wherein prompting the language model to determine the answer includes providing the language model with the prompt including the portion of the knowledge graph.

[0191] 33: A system as described in any one of clauses 25, 26, 27, 28, 29, 30, 31, or 32 further includes a prompting engineering component configured to perform a fourth operation, the fourth operation comprising: determining that the size of the query is greater than the context window size of the language model; and segmenting the query into a first query portion and a second query portion, the first query portion and the second query portion having corresponding sizes smaller than the context window size, wherein prompting the language model to determine that the answer includes providing the first query portion to the language model separately from the second query portion.

[0192] 34: A computer-implemented method for operating a chatbot, comprising: providing access to a chatbot interface to a user device, the chatbot interface being configured to receive queries to the chatbot; receiving the query from the user device via the chatbot interface; determining that an answer to the query is not included in a knowledge graph used by the chatbot; prompting a language model associated with the chatbot to determine the answer to the query, at least in part based on the fact that the answer is not included in the knowledge graph; receiving the answer to the query as output from the language model; adding the answer to the query to the knowledge graph; and providing the answer to the user device via the chatbot interface.

[0193] 35: A computer-implemented method as described in Clause 34, wherein the knowledge graph is a first knowledge graph and is associated with a first subject domain and a user account, the method further comprising: determining that the user account is associated with a second knowledge graph, the second knowledge graph being associated with a second subject domain; determining that the query is associated with the first subject domain; and determining, at least in part, to suggest the first knowledge graph rather than the second knowledge graph for the answer based on the association of the query with the first subject domain.

[0194] 36: A computer-implemented method as described in any one of clauses 34 or 35, wherein: via the chatbot, feedback data from the user device indicating that the answer is an incorrect answer to the query is received; a portion of the knowledge graph is identified for determining an answer to the query; and at least one of the portions of the knowledge graph is modified or removed from the knowledge graph.

[0195] 37: A computer-implemented method as described in any one of clauses 34, 35 or 36, further comprising: receiving a request from the user device via the chatbot interface to provide an indication of the source of the answer provided by the chatbot; identifying origin data stored in the knowledge graph indicating that the source of the answer is the language model; and providing the user via the chatbot interface the indication that the source of the answer is the language model.

[0196] 38: A computer-implemented method as described in any one of clauses 34, 35, 36 or 37, further comprising: receiving a request from the user device via the chatbot interface to provide an indication of the source of the answer provided by the chatbot; identifying origin data stored in the knowledge graph indicating that the source of the answer is the language model; and providing the user via the chatbot interface with the indication that the source of the answer is the language model.

[0197] 39: A computer-implemented method as described in any one of clauses 34, 35, 36, 37, or 38, further comprising: receiving feedback data via the chatbot interface indicating that a user is dissatisfied with the answer provided for the query; prompting the language model to determine a second answer to the query; receiving the second answer to the query as a second output from the language model; adding the second answer to the query to the knowledge graph; and providing the user with a response including the second answer to the query.

[0198] 40: A computer-implemented method as described in any one of clauses 34, 35, 36, 37, 38, or 39, further comprising: receiving, via the chatbot interface, a request for an explanation of why the answer to the query is provided; identifying a portion of the knowledge graph used to determine the answer to the query; generating, using information from the portion of the knowledge graph, an explanation of why the answer to the query is provided; and providing the explanation to the user device via the chatbot interface.

Claims

1. A computer-implemented method for operating a chatbot, comprising: Provide user devices with access to a chatbot interface, which is configured to receive queries to the chatbot; Query is received from the user device via the chatbot interface; The answer to the query is determined to be excluded from the knowledge graph used by the chatbot; Based at least in part on the fact that the answer is not included in the knowledge graph, the language model associated with the chatbot is prompted to determine the answer to the query; Receive the answer to the query as output from the language model; Add the answer to the query to the knowledge graph; and The response is provided to the user device via the chatbot interface.

2. The computer-implemented method of claim 1, wherein the knowledge graph is a first knowledge graph and is associated with a first subject domain and a user account, the method further comprising: The user account is determined to be associated with a second knowledge graph, and the second knowledge graph is associated with a second topic domain; The query is determined to be associated with the first topic domain; as well as The suggestion for the answer is determined, at least in part, based on the association between the query and the first topic domain, using the first knowledge graph rather than the second knowledge graph.

3. The computer-implemented method as described in claim 1 or 2, wherein: The chatbot receives feedback data from the user device indicating that the answer is an incorrect response to the query. Identify a portion of the knowledge graph used to determine the answer to the query; as well as Modify the portion of the knowledge graph or remove at least one of the portions from the knowledge graph.

4. The computer-implemented method as described in any one of claims 1, 2, or 3, further comprising: Receive requests from the user device via the chatbot interface to provide an indication of the source of the response provided by the chatbot; The source of the answer, as indicated by the identifier stored in the knowledge graph, is the origin data of the language model; as well as The source of the response provided to the user via the chatbot interface is the instruction from the language model.

5. The computer-implemented method as described in any one of claims 1, 2, 3, or 4, further comprising: Receive requests from the user device via the chatbot interface to provide an indication of the source of the response provided by the chatbot; The source of the answer, as indicated by the identifier stored in the knowledge graph, is the origin data of the language model; as well as The source of the response provided to the user via the chatbot interface is the instruction from the language model.

6. The computer-implemented method as described in any one of claims 1, 2, 3, 4, or 5, further comprising: The system receives feedback data via the chatbot interface indicating that the user is dissatisfied with the answer provided for the query. The language model is prompted to determine a second answer to the query; Receive the second answer to the query as a second output from the language model; Add the second answer to the query to the knowledge graph; as well as Provide the user with a response that includes the second answer to the query.

7. The computer-implemented method as described in any one of claims 1, 2, 3, 4, 5, or 6, further comprising: Receive requests for explanations regarding why the answer was provided for the query via the chatbot interface; Identify a portion of the knowledge graph used to determine the answer to the query; Using the information in the aforementioned portion of the knowledge graph, an explanation is generated regarding why the answer is provided for the query; as well as The explanation is provided to the user device via the chatbot interface.

8. A system comprising: One or more processors; and One or more computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations including: A query is obtained at one or more components of the system, which are configured to answer the query using at least one of a knowledge graph or a language model. The formal language representation of the query is used to determine whether the answer to the query is included in the knowledge graph; Based at least in part on the fact that the answer is not included in the knowledge graph, the language model is prompted to generate the answer to the query; Receive the answer to the query as output from the language model; as well as The answer to the query is added to the knowledge graph.

9. The system of claim 8, wherein the operation further comprises: Receive feedback data indicating that the answer is an incorrect answer to the query; as well as Modify the answer in the knowledge graph or remove at least one of the answers from the knowledge graph.

10. The system of claim 8 or 9, wherein the answer is a first answer to the query, and the operation further includes: The storage indicates that the language model is used to determine the origin information of the first answer; Obtain a second answer to the query from another information source; Add the second answer to the knowledge graph; as well as The knowledge graph stores second origin information that indicates the other information source for obtaining the second answer.

11. The system of any one of claims 8, 9, or 10, wherein the operation further comprises: Receive a request from a user account registered with the system to generate the knowledge graph on behalf of the user account; The language model is iteratively suggested using multiple queries; Receive the answers to the multiple queries as output from the language model; as well as The answers to the multiple queries are added to the knowledge graph.

12. The system of any one of claims 8, 9, 10 or 11, wherein the operation further comprises: The knowledge graph is identified by a portion of information related to the query. as well as Generate suggestions for the language model to determine the answer to the query, the suggestions including the portion of the knowledge graph and the query. The prompt suggests that the language model determines the answer by providing the prompt to the language model.

13. The system of any one of claims 8, 9, 10, 11, or 12, wherein the answer is a first answer to the query, and the operation further includes: The language model is prompted to determine a second answer to the query; Receive the second answer to the query as a second output from the language model; A summary of known answers to the query is generated using the first and second answers, wherein the summary of known answers includes a data size smaller than the context window of the language model; as well as The prompt includes providing the language model with a summary of the known answer to the query, indicating that the language model is determining a third answer.

14. The system of any one of claims 8, 9, 10, 11, 12, or 13, wherein the answer is a first answer and the query is a first query, and the operation further comprises: The language model is suggested to determine the second answer to the second query; Receive the second answer to the query as a second output from the language model; The second answer is determined to be associated with a concept that is not allowed to be represented in the knowledge graph; as well as Avoid adding the second answer to the knowledge graph.

15. The system of any one of claims 8, 9, 10, 11, 12, 13, or 14, wherein the query is a first instance of the query, and the operation further comprises: The system receives feedback data indicating that the response output by the language model is an incorrect answer to the query. Receive input data, the input data including modifications to the answer in the knowledge graph to produce a modified answer, the modified answer being the correct answer to the query; Receive a second instance of the query from the user account associated with the system; The answer to the modified query is determined in the knowledge graph; as well as The modified response is provided to the user account without using the language model to determine the modified response.

Images