Generating precisely referenced responses from user supplied documents by large language models

Abstract

In one embodiment, a method includes receiving a query from a client system associated with a user, accessing a knowledge graph generated based on a set of documents defined by the user, generating a response corresponding to the query by a large language model (LLM) and based on the knowledge graph, wherein the response is based on information extracted from one or more documents of the set of documents and the knowledge graph, and wherein the response comprises one or more indexes identifying where the information is extracted within the one or more documents, and sending instructions for presenting the response with the indexes to the client system responsive to the query.

Description

[0001] 072396.1102

[0002] GENERATING PRECISELY REFERENCED RESPONSES FROM USER SUPPLIED

[0003] DOCUMENTS BY LARGE LANGUAGE MODELS

[0004] CROSS-REFERENCE TO RELATED APPLICATIONS

[0005] This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 682, 113, filed August 12, 2024, the content of which is incorporated herein by reference in its entirety, and to which priority is claimed.

[0006] TECHNICAL FIELD

[0007] This disclosure generally relates to large language models.

[0008] BACKGROUND

[0009] A large language model (LLM) is a computational model notable for its ability to achieve general-purpose language generation and other natural language processing tasks such as classification. Based on language models, LLMs acquire these abilities by learning statistical relationships from vast amounts of text during a computationally intensive selfsupervised and semi-supervised training process. LLMs can be used for text generation, a form of generative artificial intelligence (Al), by taking an input text and repeatedly predicting the next token or word.

[0010] SUMMARY OF PARTICULAR EMBODIMENTS

[0011] The purpose and advantages of the disclosed subject matter will be set forth in and apparent from the description that follows, as well as will be learned by practice of the disclosed subject matter. Additional advantages of the disclosed subject matter will be realized and attained by the methods and systems particularly pointed out in the written description and claims hereof, as well as from the appended drawings.

[0012] To achieve these and other advantages, and in accordance with the purpose of the disclosed subject matter, as embodied and broadly described, the disclosed subject matter presents systems, methods, and apparatuses that can be used to generating responses for user queries. For example, certain non-limiting embodiments can be used to query large language models for preci sely-references response from user-defined documents.

[0013] 1

[0014] ACTIVE 511363720.1 072396.1102

[0015] In certain non-limiting embodiments, one or more computing systems can receive a query from a client system associated with a user. The computing systems can then access a knowledge graph generated based on a set of documents defined by the user. The computing systems can then generate, by a large language model (LLM) and based on the knowledge graph, a response corresponding to the query. In one feature, the response is based on information extracted from one or more documents of the set of documents and the knowledge graph. In another feature, the response comprises one or more indexes identifying where the information is extracted within the one or more documents. The computing systems can further send, to the client system responsive to the query, instructions for presenting the response with the one or more indexes.

[0016] In certain non-limiting embodiments, one or more computer-readable non- transitory storage media embodying software is operable when executed to receive a query from a client system associated with a user. The computer-readable non-transitory storage media embodying software is further operable when executed to access a knowledge graph generated based on a set of documents defined by the user. The computer-readable non- transitory storage media embodying software is further operable when executed to generate, by a large language model (LLM) and based on the knowledge graph, a response corresponding to the query. In one feature, the response is based on information extracted from one or more documents of the set of documents and the knowledge graph. In another feature, the response comprises one or more indexes identifying where the information is extracted within the one or more documents. The computer-readable non-transitory storage media embodying software is further operable when executed to send, to the client system responsive to the query, instructions for presenting the response with the one or more indexes.

[0017] In certain non-limiting embodiments, a system can comprise one or more processors and a non-transitory memory coupled to the processors comprising instructions executable by the processors. The processors are operable when executing the instructions to receive a query from a client system associated with a user. The processors are further operable when executing the instructions to access a knowledge graph generated based on a set of documents defined by the user. The processors are further operable when executing the instructions to generate, by a large language model (LLM) and based on the knowledge graph, a response corresponding to the query. In one feature, the response is based on information extracted from one or more documents of the set of documents and the knowledge graph. In another feature, the response comprises one or more indexes identifying where the information is extracted within the one or more documents. The processors are further operable when

[0018] 2

[0019] ACTIVE 511363720.1 072396.1102 executing the instructions to send, to the client system responsive to the query, instructions for presenting the response with the one or more indexes.

[0020] Furthermore, the disclosed embodiments of the methods, computer readable non-transitory storage media, and systems can have further non-limiting features as described below.

[0021] In certain non-limiting embodiments, the computing systems can further split the set of documents into a plurality of chunks. In one feature, splitting the set of documents is based on one or more of a splitting criterion or a machine-learning model.

[0022] In certain non-limiting embodiments, the computing systems can encode each piece of information associated with each document of the set of documents into a vector embedding, the vector embeddings capturing a semantic meaning associated with the piece of information. In one feature, each piece of information comprises a paragraph, a page, or an article.

[0023] In certain non-limiting embodiments, the computing systems can generate, by a plurality of LLMs, the knowledge graph based on the pieces of information associated with each document.

[0024] In certain non-limiting embodiments, the computing systems can generate, based on the LLM, a formalized and standardized query for the received query. The LLM can be configured to enable the user to refine and supplement their questions to ensure maximum coverage and accuracy in the search results. In one feature, the formalized and standardized query captures semantics associated with the received query. The computing systems can further generate a query vector and compare the query vector with the knowledge graph. In one embodiment, the query vector can be interpreted by a natural -language processing (NLP) model. In another embodiment, the computing system can further determine similarity between the query vector and the knowledge graph using metrics determined based on graph theory.

[0025] In certain non-limiting embodiments, the computing systems can further perform similarity searches between the query vector and embeddings of nodes of the knowledge graph. In one feature, each of the similarity search calculates a distance metric between the query vector and an embedding of a node of the knowledge graph.

[0026] In certain non-limiting embodiments, the computing systems can further identify one or more pieces of information based on the similarity searches and their associated distance metrics, wherein the identified pieces of information are within the one or more documents. The computing systems can retrieve the identified pieces of information. In one feature, the extracted information is based on the retrieved pieces of information. The

[0027] 3

[0028] ACTIVE 511363720.1 072396.1102 computing systems can further feed the retrieved pieces of information into the LLM. In one feature, the LLM is configured to synthesize the pieces of information to generate the response.

[0029] In certain non-limiting embodiments, each of the one or more indexes comprises one or more of a paragraph index, a page index, or an article index.

[0030] In certain non-limiting embodiments, the knowledge graph comprises a plurality of nodes and a plurality of edges connecting the nodes. In one feature, each of the nodes represents a concept, an object, an entity, or an action that the LLM is configured to selfidentify as instructed by a prompt selected by the user. In another feature, each of the edges represents a relationship between the nodes that edge connects.

[0031] In certain non-limiting embodiments, the set of documents are at different storage locations.

[0032] In certain non-limiting embodiments, the computing systems can generate, based on the query, a plurality of prompts configured to be inputted into the LLM to elicit a response from the LLM. In one feature, the plurality of prompts are based on different styles or formalities of a way and tone that the LLM is configured to explain a concept or answer a question. The computing systems can further send, to the client system, instructions for presenting the plurality of prompts.

[0033] In certain non-limiting embodiments, the computing systems can receive, from the client system, a user selection of a first prompt from the plurality of prompts. In one feature, the first prompt is based on a first style or formality. Accordingly, the response is based on the first style or formality.

[0034] In certain non-limiting embodiments, the query and the response are associated with a particular domain. The computing systems can access a plurality of historical responses generated by one or more humans associated with the particular domain. The computing systems can then use natural language processing (NLP) to extract one or more language registers from the historical responses. The computing systems can further generate a prompt based on the extracted language registers. The computing systems can further input the prompt to the LLM to adjust the response to match the extracted language registers.

[0035] In certain non-limiting embodiments, the language registers comprise one or more of core voice characteristics, a vocabulary and language pattern, a grammar and sentence structure, a message structure pattern, or a key behavioral pattern.

[0036] In certain non-limiting embodiments, the query and the response are associated with a particular domain. The computing systems can access a plurality of historical message exchanges associated with the particular domain. The computing systems can then identify,

[0037] 4

[0038] ACTIVE 511363720.1 072396.1102 from the historical message exchanges, one or more historical responses related to the query. The computing systems can further analyze the historical responses to determine one or more language registers of the historical responses. The computing systems can further generate a prompt based on the language registers. The computing systems can further input the prompt to the LLM to adjust the response to match the language registers.

[0039] In certain non-limiting embodiments, the language registers comprise one or more of an average length of the identified historical responses, a response style, or a crossreference to a document.

[0040] It is to be understood that both the foregoing general description and the following detailed description are exemplary and are intended to provide further explanation of the disclosed subject matter claimed. These and other features, aspects, and advantages of the disclosure will be apparent from a reading of the following detailed description together with the accompanying drawings, which are briefly described below. The invention includes any combination of two, three, four, or more of the above-noted embodiments as well as combinations of any two, three, four, or more features or elements set forth in this disclosure, regardless of whether such features or elements are expressly combined in a specific embodiment description herein. This disclosure is intended to be read holistically such that any separable features or elements of the disclosed invention, in any of its various aspects and embodiments, should be viewed as intended to be combinable unless the context clearly dictates otherwise.

[0041] BRIEF DESCRIPTION OF THE DRAWINGS

[0042] FIGS. 1A-1B illustrate an example knowledge graph.

[0043] FIGS. 2A-2B illustrate an example user interface showing an LLM-generated response with the precise locations of the data source.

[0044] FIG. 3 illustrates an example flow diagram for generating a response with language-register matching.

[0045] FIG. 4 illustrates an example method for generating a response with precise indexing by an LLM.

[0046] FIG. 5 illustrates an example computer system.

[0047] 5

[0048] ACTIVE 511363720.1 072396.1102

[0049] DETAILED DESCRIPTION

[0050] When combined with an open-source large language model (LLM), the embodiments disclosed herein may enable users to define a set of their own documents and then interact with the knowledge embodied by that set of documents through a chat-bot interface. The user-defined set of documents can be of any arbitrary size. In addition, the set of documents can be stored in any location without the need for uploading to a third-party server. The embodiments disclosed herein may specifically tailor this process to situations where it is essential to have specificity to the user-defined documents in a manner that gives thorough results with technical accuracy, conceptual synthesis, and precise references to avoid the concerns of the Al-hallucinations. The embodiments disclosed herein may further remind a user the facts and questions they might overlook during the query process.

[0051] LLMs may have a limitation referred to as Al hallucination, where an LLM generates incorrect, misleading information confidently. LLMs may have another limitation of limited memory, known as the token limits. To overcome these limitations and avoid complex prompt engineering, the embodiments disclosed herein develop a knowledge graph (KG), retrieval argument generation (RAG), and embeddings.

[0052] In particular embodiments, a computing system may firstly use a document preprocessor to identify a pattern for splitting input files into smaller chunks with labeled metadata. As an example and not by way of limitation, the labeled metadata may include page identifiers. The splitting may be based on volumes, conferences, etc.

[0053] In particular embodiments, pre-processing may be performed manually by hard coding the splitting criteria. As an example and not by way of limitation, manual pre-processing may be performed when creating a custom solution for a user with particular, well-defined patterns in document formatting. Example documents suitable for manual pre-processing may include conference proceedings or company policies.

[0054] In particular embodiments, pre-processing may be performed using a machinelearning process. The machine-learning process is used to choose the best splitting algorithm for each input file based on the training data. As an example and not by way of limitation, preprocessing based on machine learning may be performed for deployments to broader populations where there is diversity and unpredictability in the document formatting. An example may be university STEM students who may define the document sets based on course materials.

[0055] 6

[0056] ACTIVE 511363720.1 072396.1102

[0057] In particular embodiments, the computing system may perform knowledge vectorization and embedding. During this process, the computing system may convert the external knowledge base, such as a collection of documents or a database, into vector embeddings. For Each piece of information (e.g., a paragraph, a page, or an article) may be encoded into a vector using embedding techniques from the field of natural language processing. This process may capture the semantic meaning of the text in a dense vector form that the user can specify the dimensionality.

[0058] FIGA. 1A-1B illustrate an example knowledge graph 100. In particular embodiments, the computing system may further generate a knowledge graph 100 based on the raw text that the user is provided. As can be seen from FIGS. 1 A-1B, the knowledge graph 100 may comprise a plurality of nodes 110 and a plurality of edges 120 connecting the nodes 110. Each node 110 may correspond to a textual description of the knowledge base. Each node 110 may be associated with an identifier and a name. Each edge 120 may reflect the relationship between the nodes 110 the edge 120 connects.

[0059] In particular embodiments, the computing system may use the similarity search algorithm to perform a similarity search between a user query and the database comprising the knowledge embeddings. When a user query is received at the computing system, the computing system may first call a LLM model to formalize and standardize the query. In this standardization process, the LLM helps users refine and supplement their questions to ensure maximum coverage and accuracy in the search results. In particular embodiments, the embedding model may be the same as the model used for generating the knowledge database. At the same time, the standardization of the user query may enable both experienced and inexperienced users of LLMs to obtain results they desire without impacts from nuances of the language a user selects in their prompt. In alternative embodiments, the embedding model may be different from the model used for generating the knowledge database as long as the dimensionality of the query vectors and knowledge base vectors are remain the same.

[0060] To generate a response for a user query, the computing system may generate a prompt configured for eliciting outputs from the LLM model based on information associated with the query. In particular embodiments, a plurality of prompts may be predefined. Different prompts may be designed for different users. When a user submits a query, the computing system may provide the options of different prompts to the user. For example, a user may select a prompt that is based on formal language. As another example, a user may select a prompt that is based on casual language. Once a user selects a prompt, the LLM may generate a

[0061] 7

[0062] ACTIVE 511363720.1 072396.1102 response that is consistent with the style / format of the selected prompt, which is further presented to the user.

[0063] The computing system may then perform similarity searches between the query vector and the vectors from the database comprising the knowledge embeddings. Each similarity search may calculate a distance metric, for example, cosine similarity. The similarity search may be customized for each user. In particular embodiments, the similarity search may make use of the knowledge graph. The computing system may use graph search algorithms to identify nodes in the knowledge graph that are relevant to the query. In particular embodiments, the search may identify a first node and then expand from the first node to one or more other nodes. This intrinsic nature of the graph can also help the user to remind the aspect they might overlook in the query process once the first node is identified in the graph.

[0064] Based on the similarity search, the computing system may determine that particular portions (e.g., passages) of one or more documents or particular documents in the database are most semantically similar to the query. The computing system may then retrieve these top ranked results as the context or relevant information for answering the user query.

[0065] In particular embodiments, the computing system may feed the retrieved results into a large language model (LLM). The LLM may use its generative capabilities to synthesize the results, ensuring the response is coherent, grammatically correct, and contextually relevant. The LLM may also perform some degree of reasoning based on the intrinsic and unique nature of the knowledge graph or synthesis to generate a comprehensive answer that aligns with the user query.

[0066] The embodiments disclosed herein are highly scalable and have generated a large-scale database. Responsive to a user query, the embodiments disclosed herein can generate a response based on knowledge from at least 8000 pages of PDF.

[0067] FIGS. 2A-2B illustrate an example user interface 200 showing an LLM- generated response with the precise locations of the data source. FIG. 2 A shows the response 210. The LLM further identifies the location 220 of the data source, e.g., page 192, which is mapped to the original page 230 of the document illustrated in FIG. 2B. Furthermore, some of the key information 240, e.g., 215.814 ft is mapped to the original page 230 of the document, which further demonstrate the accurate indexing feature of the embodiments disclosed herein.

[0068] As previously described, the LLM may generate a response that is consistent with the style / format of the selected prompt. In certain non-limiting embodiments, the computing system may enhance the response with features of matching language registers. For

[0069] 8

[0070] ACTIVE 511363720.1 072396.1102 example, if a patient is seeking a response, the response may be adjusted to match the language register of health providers.

[0071] FIG. 3 illustrates an example flow diagram 300 for generating a response with language-register matching. The example flow diagram 300 is based on a use case of a patient seeking health / medical related responses. Upon receiving a patient question 302, the LLM 304 may perform hybrid search and re-rank 306, which may include expert-curated knowledge graph search 308, semantic search 310, and keyword search 312. The LLM 304 may further perform provider language-register matching 314 on the search results from the hybrid search and re-rank 306. Language-register tuning may be embedded as prompt engineering 316 to enable the LLM 304 to generate a response with language-register matching. In certain nonlimiting embodiments, provider language-register matching 314 leverages a machine learning pipeline on physician-patient conversations 318, enabling provider-specific and preferred tone and length. The machine learning pipeline may record user editing and learn their writing style and language register 320. The LLM 304 may then return formatted response 322 after provider language-register matching 314. Such response 322 may have the following advantages 324: copy-and-paste ready (e.g., provider preferred tone and length and comprehensive and hallucination-free (e.g., referenceable to page number). The computing system may further provide an answer 326 to patient question.

[0072] In an operation, the computing system may use an LLM to extract the important characters from health provider’s writing. The computing system may use natural language processing (NLP) to obtain some statistics from a health provider’s writing. In another operation, from many (e.g., 60,000) real message exchanges between health providers and patients, the computing system can identify the top-k most relevant questions and real physician responses related to the patient query. The computing system can then analyze the average length of those responses, as well as their tone, urgency, and level of detail. These responses can be cross-referenced with textbook answers retrieved by the embodiments disclosed herein, to highlight potentially missing points and provide them as prompts to the LLM. In certain non-limiting embodiments, the two operations described above can be executed concurrently.

[0073] The computing system may then inject the results from the aforementioned two operations into LLM through prompts to mimic the health provider’s writing style and habits, which is referred as language registers in this disclosure.

[0074] In certain non-limiting embodiments, the computing system may use the LLM to extract the following characters from health provider’s writing: core voice characteristics,

[0075] 9

[0076] ACTIVE 511363720.1 072396.1102 vocabulary and language patterns, grammar and sentence structure, message structure pattern, key behavioral patterns, etc. For example, core voice characteristics may include tone, which may be consistently polite, supportive, professional with a friendly and approachable register, using expressions of gratitude and reassurance to build rapport, remaining empathetic even when addressing delays or concerns, etc. As another example, core voice characteristics may include approach, which may be professional yet warm, empathetic, and reassuring, always maintaining a caring, supportive tone while delivering medical information clearly, etc.

[0077] For example, vocabulary and language patterns may include using medical terminology but explains complex concepts in accessible language, frequently using phrases such as “Let me know”, “I’m happy to”, “Please feel free to”, “Thank you for”, “I understand”, “I can help”, balancing professionalism with warmth through word choice, explaining medical terms parenthetically when needed, etc.

[0078] For example, grammar and sentence structure may include using complete sentences with proper punctuation, often using “I” statements to personalize communication, employing conditional phrases like “If you have any questions” or “When you’re ready”, creating conversational flow with connecting phrases, etc.

[0079] For example, message structure pattern may include opening (e.g., warm greeting using patient’s name), context (e.g., brief acknowledgment of the situation), main content (e.g., clear, organized information delivery), action items (e.g., specific next steps or recommendations), closing (e.g., reassuring tone with offer for further help), and sign-off (e.g., professional yet warm closing such as “All the best”, “-Dr. Name”).

[0080] For example, key behavioral patterns may include always addressing patients by name in greetings, acknowledging concerns before providing information, offering specific help and showing availability, using encouraging language (e.g., “doing great”, “good news”, etc.), providing clear explanations for medical decisions, referencing previous interactions for continuity, offering flexibility and options when possible.

[0081] In certain non-limiting embodiments, the computing system may structure the real message exchange between health providers and patients. For example, each structured data entry contains four columns, including department, topic, question, and answer. The computing system may perform preprocessing on the structured data. The computing system may perform data clustering by department. For example, among the 60000 messages, internal medicine may have 20318 cases, otolaryngology may have 16548 cases, nutrition and health may have 14456 cases, neonatology may have 8145 cases, surgery may have 5170 cases, ophthalmology may have 1967 cases, and orthopedics may have 1327 cases.

[0082] 10

[0083] ACTIVE 511363720.1 072396.1102

[0084] The computing system may further perform data clustering within each department. To do so, the computing system may identify the a plurality of most similar conversations for each query. For example, the number of most similar conversation can be 10, 20, 30, or any suitable number. The computing system may then determine the average length of each response, the response style, cross-references of the textbook information. The average length of each response may give the LLM a reference of the typical length to answer patients for each type of question. The response style can be directly responding to the question or providing reassurance first, which may depend on the question itself, whether it is urgent, informational, or just seeking reassurance. Cross-references of the textbook information may be used as a basis for suggested clarifying questions for the LLM when the user’s question relates to one of these topics but lacks details from the information extract from the documents.

[0085] In certain non-limiting embodiments, the computing system can verify and validate the qualities of language-register matching features. For example, the computing system may extract de-identified patient questions and doctor responses from a database, use the trained LLM to answer these same patient questions, and evaluate and compare the answers from LLM with anonymized doctor responses. The evaluation and comparison can be used to further finetune the ability of the LLM for language-register matching features by injecting the evaluation and comparison into the retraining process of the LLM.

[0086] FIG. 4 illustrates an example method 400 for generating a response with precise indexing by an LLM. The method may begin at step 410, where the computing system may receive a query from a client system associated with a user. At step 420, the computing system may access a knowledge graph generated based on a set of documents defined by the user. At step 430, the computing system may generate, by a large language model (LLM) and based on the knowledge graph, a response corresponding to the query, wherein the response is based on information extracted from one or more documents of the set of documents, and wherein the response comprises one or more indexes identifying where the information is extracted within the one or more documents. At step 440, the computing system may send to the client system responsive to the query, instructions for presenting the response. Particular embodiments may repeat one or more steps of the method of FIG. 4, where appropriate. Although this disclosure describes and illustrates particular steps of the method of FIG. 4 as occurring in a particular order, this disclosure contemplates any suitable steps of the method of FIG. 4 occurring in any suitable order. Moreover, although this disclosure describes and illustrates an example method for generating a response with precise indexing by an LLM including the particular steps of the method of FIG. 4, this disclosure contemplates any suitable method for generating a response

[0087] 11

[0088] ACTIVE 511363720.1 072396.1102 with precise indexing by an LLM including any suitable steps, which may include all, some, or none of the steps of the method of FIG. 4, where appropriate. Furthermore, although this disclosure describes and illustrates particular components, devices, or systems carrying out particular steps of the method of FIG. 4, this disclosure contemplates any suitable combination of any suitable components, devices, or systems carrying out any suitable steps of the method of FIG. 4.

[0089] FIG. 5 illustrates an example computer system 500. In particular embodiments, one or more computer systems 500 perform one or more steps of one or more methods described or illustrated herein. In particular embodiments, one or more computer systems 500 provide functionality described or illustrated herein. In particular embodiments, software running on one or more computer systems 500 performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems 500. Herein, reference to a computer system may encompass a computing device, and vice versa, where appropriate. Moreover, reference to a computer system may encompass one or more computer systems, where appropriate.

[0090] This disclosure contemplates any suitable number of computer systems 500. This disclosure contemplates computer system 500 taking any suitable physical form. As example and not by way of limitation, computer system 500 may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (such as, for example, a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile telephone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these. Where appropriate, computer system 500 may include one or more computer systems 500; be unitary or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 500 may perform without substantial spatial or temporal limitation one or more steps of one or more methods described or illustrated herein. As an example and not by way of limitation, one or more computer systems 500 may perform in real time or in batch mode one or more steps of one or more methods described or illustrated herein. One or more computer systems 500 may perform at different times or at different locations one or more steps of one or more methods described or illustrated herein, where appropriate.

[0091] 12

[0092] ACTIVE 511363720.1 072396.1102

[0093] In particular embodiments, computer system 500 includes a processor 502, memory 504, storage 506, an input / output (I / O) interface 508, a communication interface 510, and a bus 512. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

[0094] In particular embodiments, processor 502 includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor 502 may retrieve (or fetch) the instructions from an internal register, an internal cache, memory 504, or storage 506; decode and execute them; and then write one or more results to an internal register, an internal cache, memory 504, or storage 506. In particular embodiments, processor 502 may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor 502 including any suitable number of any suitable internal caches, where appropriate. As an example and not by way of limitation, processor 502 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in the instruction caches may be copies of instructions in memory 504 or storage 506, and the instruction caches may speed up retrieval of those instructions by processor 502. Data in the data caches may be copies of data in memory 504 or storage 506 for instructions executing at processor 502 to operate on; the results of previous instructions executed at processor 502 for access by subsequent instructions executing at processor 502 or for writing to memory 504 or storage 506; or other suitable data. The data caches may speed up read or write operations by processor 502. The TLBs may speed up virtual-address translation for processor 502. In particular embodiments, processor 502 may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor 502 including any suitable number of any suitable internal registers, where appropriate. Where appropriate, processor 502 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 502. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

[0095] In particular embodiments, memory 504 includes main memory for storing instructions for processor 502 to execute or data for processor 502 to operate on. As an example and not by way of limitation, computer system 500 may load instructions from storage 506 or another source (such as, for example, another computer system 500) to memory 504. Processor 502 may then load the instructions from memory 504 to an internal register or internal cache.

[0096] 13

[0097] ACTIVE 511363720.1 072396.1102

[0098] To execute the instructions, processor 502 may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor 502 may write one or more results (which may be intermediate or final results) to the internal register or internal cache. Processor 502 may then write one or more of those results to memory 504. In particular embodiments, processor 502 executes only instructions in one or more internal registers or internal caches or in memory 504 (as opposed to storage 506 or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory 504 (as opposed to storage 506 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple processor 502 to memory 504. Bus 512 may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) reside between processor 502 and memory 504 and facilitate accesses to memory 504 requested by processor 502. In particular embodiments, memory 504 includes random access memory (RAM). This RAM may be volatile memory, where appropriate. Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, where appropriate, this RAM may be singleported or multi-ported RAM. This disclosure contemplates any suitable RAM. Memory 504 may include one or more memories 504, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

[0099] In particular embodiments, storage 506 includes mass storage for data or instructions. As an example and not by way of limitation, storage 506 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc, a magneto-optical disc, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. Storage 506 may include removable or non-removable (or fixed) media, where appropriate. Storage 506 may be internal or external to computer system 500, where appropriate. In particular embodiments, storage 506 is non-volatile, solid-state memory. In particular embodiments, storage 506 includes read-only memory (ROM). Where appropriate, this ROM may be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory or a combination of two or more of these. This disclosure contemplates mass storage 506 taking any suitable physical form. Storage 506 may include one or more storage control units facilitating communication between processor 502 and storage 506, where appropriate. Where appropriate, storage 506 may include one or more storages 506. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

[0100] 14

[0101] ACTIVE 511363720.1 072396.1102

[0102] In particular embodiments, I / O interface 508 includes hardware, software, or both, providing one or more interfaces for communication between computer system 500 and one or more I / O devices. Computer system 500 may include one or more of these I / O devices, where appropriate. One or more of these I / O devices may enable communication between a person and computer system 500. As an example and not by way of limitation, an I / O device may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touch screen, trackball, video camera, another suitable I / O device or a combination of two or more of these. An I / O device may include one or more sensors. This disclosure contemplates any suitable I / O devices and any suitable I / O interfaces 508 for them. Where appropriate, I / O interface 508 may include one or more device or software drivers enabling processor 502 to drive one or more of these I / O devices. I / O interface 508 may include one or more I / O interfaces 508, where appropriate. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0103] In particular embodiments, communication interface 510 includes hardware, software, or both providing one or more interfaces for communication (such as, for example, packet-based communication) between computer system 500 and one or more other computer systems 500 or one or more networks. As an example and not by way of limitation, communication interface 510 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network. This disclosure contemplates any suitable network and any suitable communication interface 510 for it. As an example and not by way of limitation, computer system 500 may communicate with an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. As an example, computer system 500 may communicate with a wireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WLMAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless network or a combination of two or more of these. Computer system 500 may include any suitable communication interface 510 for any of these networks, where appropriate. Communication interface 510 may include one or more communication interfaces 510, where appropriate. Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.

[0104] 15

[0105] ACTIVE 511363720.1 072396.1102

[0106] In particular embodiments, bus 512 includes hardware, software, or both coupling components of computer system 500 to each other. As an example and not by way of limitation, bus 512 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT) interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBAND interconnect, a low-pin-count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or a combination of two or more of these. Bus 512 may include one or more buses 512, where appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

[0107] Herein, a computer-readable non-transitory storage medium or media may include one or more semiconductor-based or other integrated circuits (ICs) (such, as for example, field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical discs, optical disc drives (ODDs), magneto-optical discs, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM-drives, SECURE DIGITAL cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these, where appropriate. A computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate.

[0108] Herein, “or” is inclusive and not exclusive, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, “A or B” means “A, B, or both,” unless expressly indicated otherwise or indicated otherwise by context. Moreover, “and” is both joint and several, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, “A and B” means “A and B, jointly or severally,” unless expressly indicated otherwise or indicated otherwise by context.

[0109] The scope of this disclosure encompasses all changes, substitutions, variations, alterations, and modifications to the example embodiments described or illustrated herein that a person having ordinary skill in the art would comprehend. The scope of this disclosure is not limited to the example embodiments described or illustrated herein. Moreover, although this disclosure describes and illustrates respective embodiments herein as including particular components, elements, feature, functions, operations, or steps, any of these embodiments may

[0110] 16

[0111] ACTIVE 511363720.1 072396.1102 include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that a person having ordinary skill in the art would comprehend. Furthermore, reference in the appended claims to an apparatus or system or a component of an apparatus or system being adapted to, arranged to, capable of, configured to, enabled to, operable to, or operative to perform a particular function encompasses that apparatus, system, component, whether or not it or that particular function is activated, turned on, or unlocked, as long as that apparatus, system, or component is so adapted, arranged, capable, configured, enabled, operable, or operative. Additionally, although this disclosure describes or illustrates particular embodiments as providing particular advantages, particular embodiments may provide none, some, or all of these advantages.

[0112] 17

[0113] ACTIVE 511363720.1

Claims

072396.1102CLAIMSWhat is claimed is:

1. A method comprising, by one or more computing systems: receiving a query from a client system associated with a user; accessing a knowledge graph generated based on a set of documents defined by the user; generating, by a large language model (LLM) and based on the knowledge graph, a response corresponding to the query, wherein the response is based on information extracted from one or more documents of the set of documents and the knowledge graph, and wherein the response comprises one or more indexes identifying where the information is extracted within the one or more documents; and sending, to the client system responsive to the query, instructions for presenting the response with the one or more indexes.

2. The method of Claim 1, further comprising: splitting the set of documents into a plurality of chunks.

3. The method of Claim 2, wherein splitting the set of documents is based on one or more of a splitting criterion or a machine-learning model.

4. The method of Claim 1, further comprising: encoding each piece of information associated with each document of the set of documents into a vector embedding, the vector embedding capturing a semantic meaning associated with the piece of information.

5. The method of Claim 4, wherein each piece of information comprises a paragraph, a page, or an article.

6. The method of Claim 4, further comprising: generating, by a plurality of LLMs, the knowledge graph based on the pieces of information associated with each document of the set of documents.

7. The method of Claim 1, further comprising:18ACTIVE 511363720.1072396.1102 generating, based on the LLM, a formalized and standardized query for the received query, wherein the formalized and standardized query captures semantics associated with the received query, and wherein the LLM is configured to enable the user to refine and supplement their questions.

8. The method of Claim 7, further comprising: generating a query vector; and comparing the query vector with the knowledge graph.

9. The method of Claim 8, wherein the query vector is interpreted by a natural -language processing (NLP) model.

10. The method of Claim 8, further comprising: determining similarity between the query vector and the knowledge graph using metrics determined based on graph theory.

11. The method of Claim 8, further comprising: performing similarity searches between the query vector and embeddings of nodes of the knowledge graph, wherein each of the similarity search calculates a distance metric between the query vector and an embedding of a node of the knowledge graph.

12. The method of Claim 11, further comprising: identifying one or more pieces of information based on the similarity searches and their associated distance metrics, wherein the identified pieces of information are within the one or more documents; and retrieving the identified pieces of information; wherein the extracted information is based on the retrieved pieces of information.

13. The method of Claim 12, further comprising: feeding the identified pieces of information into the LLM, wherein the LLM is configured to synthesize the pieces of information to generate the response.

14. The method of Claim 1, wherein each of the one or more indexes comprises one or more of a paragraph index, a page index, or an article index.19ACTIVE 511363720.1072396.110215. The method of Claim 1, wherein the knowledge graph comprises a plurality of nodes and a plurality of edges connecting the nodes, wherein each of the nodes represents a concept, an object, an entity, or an action that the LLM is configured to self-identify as instructed by a prompt selected by the user, and wherein each of the edges represents a relationship between the nodes that edge connects.

16. The method of Claim 1, wherein the set of documents are at different storage locations.

17. The method of Claim 1, further comprising: generating, based on the query, a plurality of prompts configured to be inputted into the LLM to elicit a response from the LLM, wherein the plurality of prompts are based on different styles or formalities of a way and tone that the LLM is configured to explain a concept or answer a question; and sending, to the client system, instructions for presenting the plurality of prompts.

18. The method of Claim 17, further comprising: receiving, from the client system, a user selection of a first prompt from the plurality of prompts, wherein the first prompt is based on a first style or formality; wherein the response is based on the first style or formality.

19. The method of Claim 1, wherein the query and the response are associated with a particular domain, the method further comprising: accessing a plurality of historical responses generated by one or more humans associated with the particular domain; using natural language processing (NLP) to extract one or more language registers from the historical responses; generating a prompt based on the extracted language registers; and inputting the prompt to the LLM to adjust the response to match the extracted language registers.

20. The method of Claim 19, wherein the language registers comprise one or more of core voice characteristics, a vocabulary and language pattern, a grammar and sentence structure, a message structure pattern, or a key behavioral pattern.20ACTIVE 511363720.1072396.110221. The method of Claim 1, wherein the query and the response are associated with a particular domain, the method further comprising: accessing a plurality of historical message exchanges associated with the particular domain; identifying, from the historical message exchanges, one or more historical responses related to the query; analyzing the historical responses to determine one or more language registers of the historical responses; generating a prompt based on the language registers; and inputting the prompt to the LLM to adjust the response to match the language registers.

22. The method of Claim 21, wherein the language registers comprise one or more of an average length of the identified historical responses, a response style, or a cross-reference to a document.

23. One or more computer-readable non-transitory storage media embodying software that is operable when executed to: receive a query from a client system associated with a user; access a knowledge graph generated based on a set of documents defined by the user; generate, by a large language model (LLM) and based on the knowledge graph, a response corresponding to the query, wherein the response is based on information extracted from one or more documents of the set of documents and the knowledge graph, and wherein the response comprises one or more indexes identifying where the information is extracted within the one or more documents; and send, to the client system responsive to the query, instructions for presenting the response with the one or more indexes.

24. The media of Claim 23, wherein the software is further operable when executed to: split the set of documents into a plurality of chunks.

25. The media of Claim 4124, wherein splitting the set of documents is based on one or more of a splitting criterion or a machine-learning model.

26. The media of Claim 23, wherein the software is further operable when executed to:21ACTIVE 511363720.1072396.1102 encode each piece of information associated with each document of the set of documents into a vector embedding, the vector embedding capturing a semantic meaning associated with the piece of information.

27. The media of Claim 26, wherein each piece of information comprises a paragraph, a page, or an article.

28. The media of Claim 26, wherein the software is further operable when executed to: generate, by a plurality of LLMs, the knowledge graph based on the pieces of information associated with each document of the set of documents.

29. The media of Claim 23, wherein the software is further operable when executed to: generate, based on the LLM, a formalized and standardized query for the received query, wherein the formalized and standardized query captures semantics associated with the received query, and wherein the LLM is configured to enable the user to refine and supplement their questions.

30. The media of Claim 29, wherein the software is further operable when executed to: generate a query vector; and compare the query vector with the knowledge graph.

31. The media of Claim 30, wherein the query vector is interpreted by a natural -language processing (NLP) model.

32. The media of Claim 30, wherein the software is further operable when executed to: determine similarity between the query vector and the knowledge graph using metrics determined based on graph theory.

33. The media of Claim 30, wherein the software is further operable when executed to: perform similarity searches between the query vector and embeddings of nodes of the knowledge graph, wherein each of the similarity search calculates a distance metric between the query vector and an embedding of a node of the knowledge graph.

34. The media of Claim 33, wherein the software is further operable when executed to:22ACTIVE 511363720.1072396.1102 identify one or more pieces of information based on the similarity searches and their associated distance metrics, wherein the identified pieces of information are within the one or more documents; and retrieve the identified pieces of information; wherein the extracted information is based on the retrieved pieces of information.

35. The media of Claim 34, wherein the software is further operable when executed to: feed the retrieved pieces of information into the LLM, wherein the LLM is configured to synthesize the pieces of information to generate the response.

36. The media of Claim 23, wherein each of the one or more indexes comprises one or more of a paragraph index, a page index, or an article index.

37. The media of Claim 23, wherein the knowledge graph comprises a plurality of nodes and a plurality of edges connecting the nodes, wherein each of the nodes represents a concept, an object, an entity, or an action that the LLM is configured to self-identify as instructed by a prompt selected by the user, and wherein each of the edges represents a relationship between the nodes that edge connects.

38. The media of Claim 23, wherein the set of documents are at different storage locations.

39. The media of Claim 23, wherein the software is further operable when executed to: generate, based on the query, a plurality of prompts configured to be inputted into theLLM to elicit a response from the LLM, wherein the plurality of prompts are based on different styles or formalities of a way and tone that the LLM is configured to explain a concept or answer a question; and send, to the client system, instructions for presenting the plurality of prompts.

40. The media of Claim 39, wherein the software is further operable when executed to: receive, from the client system, a user selection of a first prompt from the plurality of prompts, wherein the first prompt is based on a first style or formality; wherein the response is based on the first style or formality.23ACTIVE 511363720.1072396.110241. The media of Claim 23, wherein the query and the response are associated with a particular domain, wherein the software is further operable when executed to: access a plurality of historical responses generated by one or more humans associated with the particular domain; use natural language processing (NLP) to extract one or more language registers from the historical responses; generate a prompt based on the extracted language registers; and input the prompt to the LLM to adjust the response to match the extracted language registers.

42. The media of Claim 41, wherein the language registers comprise one or more of core voice characteristics, a vocabulary and language pattern, a grammar and sentence structure, a message structure pattern, or a key behavioral pattern.

43. The media of Claim 23, wherein the query and the response are associated with a particular domain, wherein the software is further operable when executed to: access a plurality of historical message exchanges associated with the particular domain; identify, from the historical message exchanges, one or more historical responses related to the query; analyze the historical responses to determine one or more language registers of the historical responses; generate a prompt based on the language registers; and input the prompt to the LLM to adjust the response to match the language registers.

44. The media of Claim 43, wherein the language registers comprise one or more of an average length of the identified historical responses, a response style, or a cross-reference to a document.

45. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to: receive a query from a client system associated with a user; access a knowledge graph generated based on a set of documents defined by the user;24ACTIVE 511363720.1072396.1102 generate, by a large language model (LLM) and based on the knowledge graph, a response corresponding to the query, wherein the response is based on information extracted from one or more documents of the set of documents and the knowledge graph, and wherein the response comprises one or more indexes identifying where the information is extracted within the one or more documents; and send, to the client system responsive to the query, instructions for presenting the response with the one or more indexes.

46. The system of Claim 45, wherein the processors are further operable when executing the instructions to: split the set of documents into a plurality of chunks.

47. The system of Claim 4641, wherein splitting the set of documents is based on one or more of a splitting criterion or a machine-learning model.

48. The system of Claim 45, wherein the processors are further operable when executing the instructions to: encode each piece of information associated with each document of the set of documents into a vector embedding, the vector embedding capturing a semantic meaning associated with the piece of information.

49. The system of Claim 48, wherein each piece of information comprises a paragraph, a page, or an article.

50. The system of Claim 48, wherein the processors are further operable when executing the instructions to: generate, by a plurality of LLMs, the knowledge graph based on the pieces of information associated with each document of the set of documents.

51. The system of Claim 45, wherein the processors are further operable when executing the instructions to: generate, based on the LLM, a formalized and standardized query for the received query, wherein the formalized and standardized query captures semantics associated with the25ACTIVE 511363720.1072396.1102 received query, and wherein the LLM is configured to enable the user to refine and supplement their questions.

52. The system of Claim 51, wherein the processors are further operable when executing the instructions to: generate a query vector; and compare the query vector with the knowledge graph.

53. The system of Claim 52, wherein the query vector is interpreted by a natural -language processing (NLP) model.

54. The system of Claim 52, wherein the processors are further operable when executing the instructions to: determine similarity between the query vector and the knowledge graph using metrics determined based on graph theory.

55. The system of Claim 52, wherein the processors are further operable when executing the instructions to: perform similarity searches between the query vector and embeddings of nodes of the knowledge graph, wherein each of the similarity search calculates a distance metric between the query vector and an embedding of a node of the knowledge graph.

56. The system of Claim 55, wherein the processors are further operable when executing the instructions to: identify one or more pieces of information based on the similarity searches and their associated distance metrics, wherein the identified pieces of information are within the one or more documents; and retrieve the identified pieces of information; wherein the extracted information is based on the retrieved pieces of information.

57. The system of Claim 56, wherein the processors are further operable when executing the instructions to: feed the retrieved pieces of information into the LLM, wherein the LLM is configured to synthesize the pieces of information to generate the response.26ACTIVE 511363720.1072396.110258. The system of Claim 45, wherein each of the one or more indexes comprises one or more of a paragraph index, a page index, or an article index.

59. The system of Claim 45, wherein the knowledge graph comprises a plurality of nodes and a plurality of edges connecting the nodes, wherein each of the nodes represents a concept, an object, an entity, or an action that the LLM is configured to self-identify as instructed by a prompt selected by the user, and wherein each of the edges represents a relationship between the nodes that edge connects.

60. The system of Claim 45, wherein the set of documents are at different storage locations.

61. The system of Claim 45, wherein the processors are further operable when executing the instructions to: generate, based on the query, a plurality of prompts configured to be inputted into the LLM to elicit a response from the LLM, wherein the plurality of prompts are based on different styles or formalities of a way and tone that the LLM is configured to explain a concept or answer a question; and send, to the client system, instructions for presenting the plurality of prompts.

62. The system of Claim 61, wherein the processors are further operable when executing the instructions to: receive, from the client system, a user selection of a first prompt from the plurality of prompts, wherein the first prompt is based on a first style or formality; wherein the response is based on the first style or formality.

63. The system of Claim 45, wherein the query and the response are associated with a particular domain, wherein the processors are further operable when executing the instructions to: access a plurality of historical responses generated by one or more humans associated with the particular domain; use natural language processing (NLP) to extract one or more language registers from the historical responses; generate a prompt based on the extracted language registers; and27ACTIVE 511363720.1072396.1102 input the prompt to the LLM to adjust the response to match the extracted language registers.

64. The system of Claim 63, wherein the language registers comprise one or more of core voice characteristics, a vocabulary and language pattern, a grammar and sentence structure, a message structure pattern, or a key behavioral pattern.

65. The system of Claim 45, wherein the query and the response are associated with a particular domain, wherein the processors are further operable when executing the instructions to: access a plurality of historical message exchanges associated with the particular domain; identify, from the historical message exchanges, one or more historical responses related to the query; analyze the historical responses to determine one or more language registers of the historical responses; generate a prompt based on the language registers; and input the prompt to the LLM to adjust the response to match the language registers.

66. The system of Claim 65, wherein the language registers comprise one or more of an average length of the identified historical responses, a response style, or a cross-reference to a document.28ACTIVE 511363720.1

Images