Processing natural language queries for network-based services

Abstract

Systems and methods are provided for a natural language question answering service to provide answers to natural language questions with respect to network-based services or computing domains. A natural language question answering service may receive a natural language question from a client computing device. An aggregator of a natural language question answer service may retrieve paragraphs from a search system and generate prompts based on questions. A Large Language Model (LLM) of a natural language question answer service may receive cues and provide answers. The answer may be verified by a verifier of the natural language question answering service. Attributes may be applied to answers and retrieved paragraphs to generate references, embedded references, and similar questions. A watermarking module of the natural language question answering service can watermark the answer under the condition that the answer is verified.

Description

[0001] This application claims U.S. Patent Application No. 18 / 344,698, filed June 29, 2023, entitled “Proceeding Natural Language Queries for Network-Based Services”; U.S. Patent Application No. 18 / 344,707, filed June 29, 2023, entitled “Proceeding Natural Language Queries with Attrition Enrichment”; U.S. Patent Application No. 18 / 344,756, filed June 29, 2023, entitled “Proceeding Natural Language Queries with Verification and Watermarking”; and U.S. Patent Application No. 18 / 344,747, filed June 29, 2023, entitled “Proceeding Natural Language Queries with API Calls and API”. The rights of “EXECUTIONS”, the disclosures of which are incorporated herein by reference in their entirety and used for all purposes. Background Technology

[0002] Generally, computing devices and communication networks can be used to exchange data or information. In common applications, a computing device can request content from another computing device via a communication network. For example, a client with access to a computing device can use a software application to request content from a server computing device via a network (e.g., the Internet). In such embodiments, the client's computing device may be referred to as a client computing device, and the server computing device may be referred to as a content provider.

[0003] In some applications, web service providers can instantiate various web-based services that handle client requests for data. For example, web services associated with query processing or question-answering assistants (such as chatbots) can correspond to web-based services that involve human interaction to provide information. Attached Figure Description

[0004] Embodiments of various inventive features will now be described with reference to the following accompanying drawings. Throughout the drawings, reference numerals may be used repeatedly to indicate the correspondence between referred elements. The drawings are provided to illustrate exemplary embodiments described herein and are not intended to limit the scope of this disclosure. For ease of identification of any particular element or action discussed, the highest significant digit in the reference numerals generally refers to the drawing number in which the element is first introduced.

[0005] Figure 1 A block diagram depicts an environment for a natural language question answering service according to various aspects of this application, the environment including one or more client computing devices and network services, the network services including a search system component, an aggregator component, a user context component, a large language model (LLM) component, a validator component, and a watermarking component; Figure 2A This is a block diagram illustrating the architecture of an aggregator component for retrieving paragraphs and question-and-answer (QA) pairs from a search system based on natural language questions, according to various aspects of this application; Figure 2B This is a block diagram illustrating the architecture of a validator component for validating answers from a large language model (LLM) based on various aspects of this application. Figure 3 Based on all aspects of this application Figure 1 A block diagram of the environment, depicting the illustrative interactions of the different components of a natural language question-answering service based on the analysis of natural language questions; and Figure 4 This is a flowchart illustrating the routine of answering questions in a natural language question-answering service according to various aspects of this application. Detailed Implementation

[0006] Various aspects of this disclosure relate to systems and methods for providing web-based services for natural language problem processing. More specifically, one or more aspects of this application may include web-based services for processing natural language queries (e.g., questions, prompts, commands, etc.) provided by computing devices to supplement, optimize, or otherwise modify the natural language queries. One or more aspects of this application may include web-based services for further processing natural language queries based on the processed query and processing result verification using LLM-based processing resources.

[0007] Generative artificial intelligence (AI) models (e.g., Large Language Models (LLMs), question-answering assistants, chatbots, etc.) can sometimes accurately answer natural language questions (e.g., provide answers to questions that don't require follow-up questions), but they are known to have problems. For example, they tend to be trained on large amounts of internet data extracted from a specific period (e.g., 2017–2019). Therefore, models not trained on recent internet data lack the ability to answer questions about that recent internet data (e.g., who won the 2021 NBA Finals?). Furthermore, because generative AI models are trained on public internet data, they lack the ability to answer questions about private data (e.g., information specific to and about internal troubleshooting guides for web-based storage service providers, frequently asked questions (FAQs), etc.). In addition, natural language questions themselves have their own set of problems. For example, no matter how accurate a generative AI model is, natural language queries may be entered in some way (e.g., flawed, insufficiently broad, etc.) (e.g., word choice, word order, etc.) such that the answer generated by the generative AI model is (i) incomplete or incorrect relative to the question, or (ii) fails to capture the intent of the question writer. This can lead to inefficient use of the generative AI model's resources (e.g., multiple question and answer rounds may not yield an answer that the entity that entered the question would agree with). Furthermore, even if the natural language question is not erroneous or overly narrow, the generative AI model may still generate answers that are determined to be incorrect or wrong (e.g., not truly answering the question, the answer not supported by the generative AI's training data, the answer generated by the generative AI model differing from expectations, etc.), a phenomenon known as "illusion." Incorrectly generated answers can occur when asking questions about specific services based on private networks, because the public internet lacks this private information to train generative AI models.

[0008] To address at least some of the aforementioned deficiencies, a natural language question answering service may implement one or more modules to answer natural language questions about specific web-based services and / or computing domains. The natural language question answering service can provide detailed answers (e.g., processing results) about web-based services and computing domains by referring to a search system that has question-and-answer (QA) pairs specifically generated for question answering services. Furthermore, the natural language question answering service can utilize stored information to reformulate the natural language question, thereby generating an answer that is more accurate than the original reformulation.

[0009] Illustratively, a natural language question-and-answer service may utilize an aggregator to retrieve paragraphs (and their corresponding question-and-answer pairs (QA pairs)) based on a natural language question to modify, update, or supplement the natural language question, and generate suggestions. The aggregator may analyze the natural language question and determine from which search systems the paragraphs and QA pairs to be retrieved to answer the question. The aggregator may retrieve those paragraphs (and associated QA pairs) and use them, along with the question, to formulate suggestions. For the purposes of this application, a suggestion may correspond to (e.g., from all retrieved paragraphs) a small selection of paragraphs and the QA pairs of those selected paragraphs, along with the natural language question (e.g., in the form of a question, where the question may be a suggestion or a command).

[0010] In some embodiments, a natural language question-and-answer service may utilize a trained large language model (LLM) to use prompts and generate answers to natural language questions. The LLM may be a trained machine learning model that uses retrieval-enhanced generation (RAG) techniques to (e.g., additional or alternative vocabulary techniques) semantically generate answers to the question. The LLM may be trained, at least on QA pairs from search systems associated with web-based service providers (e.g., web-based storage services, web-based analytics, etc.), enabling the LLM to generate answers specific to those web-based service providers.

[0011] Illustratively, the natural language question answering service can utilize a validator to verify the answer generated by the LLM to ensure it is not erroneously generated (e.g., an illusion). The validator can utilize one or more modules to ensure the answer is not erroneously generated. The validator's text overlap module determines how much text overlaps between the answer and the retrieved paragraphs. The validator's textual natural language inference (NLI) module can determine whether the answer generated from the LLM contradicts the retrieved paragraphs from the aggregator by using premises and assumptions. The validator's relational NLI module can determine whether the answer generated from the LLM is erroneously generated regarding the natural language question by using head, tail, and relation triples. Furthermore, the validator's membership inference attack module can determine whether the question is similar to previous questions and whether it is in the LLM's training set.

[0012] In some embodiments, a natural language question answering service may use a watermarking component to watermark verified answers from an LLM (Local Language Management Machine). The watermarking component may modify (e.g., regenerate) the verified answers so that they are specific to the natural language question answering service. In some embodiments, the watermark may include a hidden pattern in the regenerated verified answers that is imperceptible to humans, while allowing the hidden pattern to be recognized as synthetic by the system (e.g., the natural language question answering service) through an algorithm.

[0013] Although aspects of this disclosure are described relative to exemplary network components, interactions, and routines, those skilled in the art will recognize that one or more aspects of this disclosure may be implemented in various environments, system architectures, client computing device architectures, etc. Similarly, references to specific devices (such as client computing devices) should be considered general references and are not intended to provide additional meaning or configuration for individual client computing devices. Furthermore, the examples are intended to be illustrative in nature only and should not be construed as limiting.

[0014] Figure 1 A block diagram depicting an embodiment of environment 100 is provided. Environment 100 may include a network 116 connecting multiple client computing devices 122 to one or more network-based services, illustratively, a natural language question answering service 102. Furthermore, the natural language question answering service 102 may receive natural language questions (e.g., about a network-based service) from client computing devices 122 and may answer the natural language question using an aggregator component 104, a user context component 105, a large language model (LLM) component 106, a validator component 108, an attribution component 109, a watermarking component 110, and a training dataset 112. The natural language question answering service 102 illustratively includes (or has access to) one or more search systems 124. Furthermore, aspects associated with the natural language question answering service 102 may be implemented as one or more components associated with one or more functions, services, machine learning models, and other components. Components may correspond to software modules implemented or executed by one or more client computing devices, which may be separate, independent client computing devices. Therefore, the components of the natural language question answering service 102 should be considered as a logical representation of the service, without requiring any specific implementation on one or more client computing devices.

[0015] like Figure 1The network 116 depicts devices and modules that connect to the system. The network can connect any number of devices. Network 116 can be a personal area network (PAN), local area network (LAN), wide area network (WAN), over-the-air broadcast network (e.g., for radio or television), cable network, satellite network, cellular telephone network, or a combination thereof. As another example, network 116 can be a publicly accessible network of linked networks that may be operated by various different parties, such as the Internet. In some embodiments, network 116 can be a private or semi-private network, such as a corporate or university intranet. Network 116 can include one or more wireless networks, such as a Global System for Mobile Communications (GSM) network, a Code Division Multiple Access (CDMA) network, a Long Term Evolution (LTE) network, or any other type of wireless network. Network 116 can use protocols and components for communication via the Internet or any of the aforementioned types of networks. For example, protocols used by network 116 may include Hypertext Transfer Protocol (HTTP), HTTP Secure (HTTPS), Message Queuing Telemetry Transport (MQTT), Constrained Application Protocol (CoAP), and so on. Protocols and components used for communication via the Internet or any of the aforementioned types of communication networks are well known to those skilled in the art and therefore are not described in more detail herein.

[0016] In some embodiments, the natural language question answering service 102 responds to queries (e.g., questions, hints, commands, etc.) submitted by a client computing device 122 via network 116 to generate processing results (e.g., answers). In other embodiments, the natural language question answering service 102 retrieves paragraphs from a search system via network 116 to provide answers to questions (e.g., in the form of natural language questions, hints, commands, etc.). The natural language question answering service 102 may utilize machine learning algorithms (such as generative AI model algorithms) to provide answers to natural language questions (e.g., information about a specific web-based service, paragraphs associated with a web-based service, links to web pages, application programming interface (API) calls, etc.) for web-based services (e.g., web-based computing resources, web-based on-demand code execution, web-based analytics, web-based storage, web-based databases, web-based AI, etc.) and / or specific computer domains (e.g., websites dedicated to social media software for sharing images, websites of retail service providers dedicated to selling products and services). For example, the natural language question answering service 102 may access different search systems (e.g., search system 124), which may store paragraphs and question-and-answer pairs (QA pairs) about a specific web-based service (e.g., question-and-answer pairs for a specific web-based storage product) or computing domain (e.g., a fashion designer's website). The natural language question answering service 102 may include a trained LLM trained at least on QA pairs from search system 124 to provide answers to questions and suggestions. Therefore, the components of the natural language question answering service 102 should be considered as a logical representation of the service, without requiring any specific implementation on one or more client computing devices.

[0017] Figure 1 The client computing device 122 can be connected to the natural language question answering service 102 via network 116. The client computing device 122 can send natural language questions (e.g., input from a user via the user interface (UI) of the client computing device 122) to the natural language question answering service 102 and receive answers from the natural language question answering service 102 based on those natural language questions. The client computing device 122 may be configured to have at least one processor. The processor can communicate with memory to maintain computer-executable instructions. The client computing device 122 can be physical or virtual. The client computing device 122 can be a mobile device, a personal computer, a server, or other type of device. The client computing device 122 has a display and input devices through which the user interacts with user interface components.

[0018] Figure 1The search system 124 can be connected to the natural language question answering service 102 via network 116. The search system 124 may be part of the natural language question answering service 102 (e.g., a service provided by the natural language question answering service 102) or associated with the natural language question answering service 102. Alternatively, the search system 124 may also be hosted outside of network 116 (e.g., a third-party service). The search system 124 may be multiple search systems that can provide (but are not limited to) paragraphs, documents, and QA pairs to the natural language question answering service 102. The natural language question answering service 102 can obtain this information and formulate answers to natural language questions related to the web-based service and / or the computer domain. For example, the search system 124 may be a data store containing frequently asked questions (FAQs) (and associated answers) about a specific web-based service (e.g., web-based storage or database).

[0019] Aggregator component 104 can retrieve paragraphs from search system 124 based on a natural language question and create hints for LLM component 106. For example, aggregator component 104 can analyze the natural language question using string matching techniques (e.g., partial string matching, dense paragraph retrieval, etc.) to determine the meaning of the natural language question. After determining the meaning of the natural language question, aggregator component 104 can then determine which part of search system 124 it can retrieve paragraphs from (e.g., from a web-based storage service QA pair system, but not from a web-based AI service QA pair system). Based on the paragraphs retrieved from search system 124 (e.g., documents, document text, document images, or document videos, etc.), aggregator component 104 can create hints for LLM component 106 to answer, where the hints from aggregator component 104 to LLM component 106 can contain some of the retrieved paragraphs and the form of the natural language question. Aggregator component 104 can be a machine learning model trained on retrieval augmentation generation (RAG) techniques.

[0020] User context component 105 may optionally capture and store user contexts of at least the users associated with client computing device 122. User context component 105 may share user contexts with LLM component 106 (e.g., or aggregator component 104 may share user contexts along with other information with LLM component 106) along with questions and related paragraphs from aggregator component 104. For example, user context component 105 may capture context information about user accounts related to network-based services. In this example, the user context captured by the user context component 105 can be: information such as the type of service the user subscribes to (e.g., the user subscribes to web-based storage services but not web-based AI services); system help information accessed regarding web-based services or any of the search systems in system 124 (e.g., the user accessed a FAQ page about web-based analytics services, the user registered a ticket with IT related to web-based database services, the user spoke with the help support hotline about web-based on-demand computing services, etc.); billing-related information associated with the user (e.g., which web-based services the user subscribes to and what premium services the user subscribes to regarding those web-based services, etc.); geographic location information associated with the user (e.g., where the user accessed web-based services, the residential location of the user who provided the web-based services, etc.); and how the user utilizes web-based services, such as by accessing telemetry information (e.g., the average or mean time the user spends on different web-based services, which services within the web-based services the user uses, such as the user using generative AI models but not linear regression AI models, etc.). User context component 105 may retrieve or receive information about usage associated with user context information from client computing device 122, natural language question answering service 102, or external systems or services (e.g., third-party providers that track user context by receiving user context details, organizing those user context details, and sending them to natural language question answering service 102 or user context component 105).

[0021] LLM component 106 may receive prompts from aggregator component 104 and user context (optionally) from user context component 105, and generate one or more answers based on the prompts and user context. LLM component 106 may be trained on at least a QA pair generated from the knowledge graph of a client of search system 124 or a web-based service (e.g., a serverless web-based on-demand computing service, etc.). LLM component 106 may take the prompts and user context received from aggregator component 104 and use a generative AI model (e.g., Retrieval Augmentation Generation (RAG) using a Natural Language Processing (NLP) architecture) to determine the answer to the natural language question. For example, if a client computing device 122 sends a natural language question about how to set up a certain type of web-based storage, LLM component 106 may use a trained generative AI model (e.g., trained on a QA pair from a web-based storage service and a client knowledge graph) to determine the answer based on the prompts received from aggregator component 104 (e.g., where the answer provides instructions and potential API calls for setting up the web-based storage).

[0022] Validator component 108 can verify that one or more answers from LLM component 106 are not incorrectly generated (e.g., no question answered, answer not supported by the training data of LLM component 106, answer generated by LLM component 106 different from the expected answer, etc.). Validator component 108 can utilize various techniques (e.g., depollution models, Natural Language Inference (NLI) triples, membership inference attacks, etc.) to determine whether an answer is incorrectly generated (also referred to as "illusion" and / or one of its derivatives). For example, validator component 108 can utilize a depollution model that filters out obscene and / or blasphemous language so that such language is not provided in the answer.

[0023] Attribution component 109 can provide various functions related to the search result information utilized in the aggregation function of aggregator component 104. Illustratively, these functions may include references to retrieved paragraphs or other source identifiers, embedded references to sentences in the answer derived from the retrieved paragraphs, and providing questions similar to the original question posed. For example, attribution component 109 can provide links (e.g., Uniform Resource Identifiers (URIs)) or titles as references to documents retrieved by aggregator component 104 (e.g., links to web pages, online documents, images, audio, video, etc.). As another example, attribution component 109 can take the answer provided by LLM component 106 and embed references to any sentence directly retrieved from any of the paragraphs retrieved by aggregator component 104 (e.g., verbatim). As yet another example, attribution component 109 can provide questions similar to the current natural language question posed (e.g., if the current question is about creating a web-based bucket, a similar question is about reading from a web-based bucket).

[0024] Watermarking component 110 can watermark one or more verified answers by generating or adding patterns (e.g., unreadable to humans) to the verified answers, enabling natural language question answering service 102 to recognize the text of the one or more answers. For example, watermarking component 110 can regenerate (e.g., embed signals into the answer) one or more answers with word combinations (e.g., patterns unreadable to humans), leaving a footprint on the word combinations that can be used to identify natural language question answering service 102 as a generator of watermarked answers (e.g., rephrased answers, where the rephrased answer has the same content as the original answer, but can be identified as being generated by natural language question answering service 102 by examining a small segment of words in the answer). Therefore, in this case, the watermark can be used to at least determine whether the answer from natural language question answering service 102 has been plagiarized.

[0025] Training dataset 112 may contain machine learning data trained on any one of the aggregator component 104, LLM component 106, validator component 108, attribution component 109, or watermarking component 110. Training dataset 112 may also contain trained data on QA pairs from one of the search systems 124, enabling the trained data to identify the types of relevant paragraphs for certain types of natural language questions. For example, LLM component 106 may utilize training dataset 112 to determine whether relevant paragraphs and QA pairs are sufficient to generate answers to natural language questions. Training dataset 112 may be periodically updated with new QA pairs. Furthermore, training dataset 112 may also contain information about customers (e.g., subscribed services, usage activity, preferences, etc.) through a knowledge graph of a web-based service or computational domain.

[0026] Figure 2A An embodiment of the architecture of an illustrative aggregator component 104 of a natural language question answering service 102 is described herein. As described herein, aggregator component 104 may be configured to determine relevant paragraphs from search systems 124 (e.g., specific web-based services and / or computer domains, etc.) based on natural language questions. As described below, aggregator component 104 may utilize string matching techniques (e.g., partial string matching, dense paragraph retrieval (DPR), etc.) to determine the meaning of the natural language question. Aggregator component 104 may obtain this meaning of the question and use it to determine which search systems 124 to retrieve paragraphs from. Additionally, aggregator component 104 may then retrieve paragraphs from the determined search systems 124 and create suggestions using the paragraphs and the natural language question (e.g., or the form of the question). As described herein, aggregator component 104 may be implemented using machine learning models (e.g., generative AI models, etc.) and RAG techniques.

[0027] Network interface 208 can provide access to, for example, Figure 1 The processing unit 206 is connected to one or more networks or computing systems via network 116. The processing unit 206 can therefore receive information and instructions from other computing systems or services via the network. The processing unit 206 can also communicate with memory 214 and also provide output information for an optional display (e.g., via a UI) via input / output device interface 212. In some embodiments, the aggregator component 104 may include a... Figure 2A The components shown are those with more (or fewer) components.

[0028] Memory 214 may include computer program instructions that the processing unit 206 executes to implement one or more embodiments. Memory 214 typically includes RAM, ROM, or other persistent or non-transitory memory. Memory 214 may store an operating system 217 that provides computer program instructions used by the processing unit 206 in the general management and operation of the aggregator component 104. Memory 214 may also include computer program instructions and other information for implementing aspects of this disclosure. For example, in one embodiment, memory 214 includes interface software 216 for communicating with other components and / or modules of the natural language question answering service 102 to generate prompts based on paragraphs and natural language questions.

[0029] As described herein, aggregator component 104 may use partial string matching (e.g., also known as “fuzzy string matching”) techniques to determine meaning from natural language questions. As used in this disclosure, partial string matching can be a technique for finding strings that partially match but do not completely match a given string. The algorithm behind partial string matching may not simply look at the equivalence of two strings, but rather quantify how close the two strings are to each other. This can be achieved using a distance metric called “edit distance,” which determines how close two strings are by identifying the minimum change required to transform one string into another. Different types of edit distance that can be used are Levenshtein distance, Hamming distance, Jaro distance, etc. For example, a user of client computing device 122 might misspell a word (type “Londin” instead of “London”) or enter a part of a word (e.g., as part of a natural language question) into a user interface connected to natural language question answering service 102; in this example, partial string matching techniques could help find the correct word, phrase, or term.

[0030] Once the meaning of the natural language question is determined, the aggregator component 104 can then determine which search systems 124 should be used to retrieve paragraphs from them. For example, if the aggregator component 104 determines that the natural language question pertains to a query about how to download content from a web-based storage service, then the aggregator component 104 can retrieve paragraphs from search systems 124 that are associated with or are web-based storage services. As another example, if the aggregator component 104 determines that the natural language question pertains to a query about how to use a function (e.g., a function for remotely performing payroll calculations) in a web-based on-demand code execution system, then the aggregator component 104 can retrieve paragraphs (and QA pairs) related to that function from search systems 124 that are web-based on-demand code execution systems or associated with web-based on-demand code execution systems.

[0031] Additionally, the memory 210 may include an Open Search Indexing module 218 for determining which documents from the retrieved paragraphs are relevant to the natural language question. In some embodiments, the Open Search Indexing module 218 analyzes the retrieved documents and performs surface matching between the documents and the natural language question. In some embodiments, the Open Search Indexing module 218 may perform surface matching by detecting or searching for keyword and phrase-level similarity between a pair of natural language content (e.g., a pair of retrieved paragraphs and a natural language question). For example, if content 1 might state “network-based services provide cloud computing service,” and content 2 might state “network-based services offers reliable, scalable, and inexpensive cloud computing services,” then the surface similarity would be the presence of “network-based service” and “cloud computing service.” As another example, one hundred documents may have been retrieved from different search systems 124, and surface matching may have determined that ten of those documents are relevant. Therefore, the Open Search Index module 218 can be used to identify all documents in a web-based service system (e.g., a web-based repository of documents related to using web-based storage services).

[0032] Additionally, memory 210 may include a dense retrieval module 219 for determining relevant portions of each document retrieved from search system 124. The dense retrieval module 219 can be created using a dense embedding (encoder) model designed to capture (e.g., from search system 124) the most salient semantic portions of each document retrieved and transform them into a fixed-dimensional dense representation that can be used to create a matrix of fixed-dimensional vectors. To perform retrieval, the dense retrieval module 219 can transform a natural language question into a similar dense embedding and find the K nearest neighbors from the index matrix. Therefore, the dense retrieval module 219 can be used to identify dense segments of text in web-based services (e.g., web-based services that store crowdsourced answers about how to use and debug web-based analytics services).

[0033] Aggregator component 104 can use multiple search systems 124 to retrieve paragraphs and can also receive QA pairs. For example, but not limited to, the search system can be a web-based system that contains documents describing how to use web-based services (e.g., a manual on how to use web-based artificial intelligence and machine learning services via a graphical user interface (GUI)). Aggregator component 104 can identify QA pairs associated with a particular type of question by tagging the title of each document and associating each title with a question (where the title is the answer to the question).

[0034] As another example, the search system could be a web-based system (or associated with a web-based system) containing web pages written by support professionals for a web-based service to address the primary concerns of the service's customers (e.g., for computing, storage, analytics, AI, etc.). In this system, each web page could detail how it addresses a problem related to the functionality of the web-based service (e.g., a technical issue). Aggregator component 104 could leverage Python functions (e.g., BeautifulSoup) to crawl the web pages to extract QA pairs (e.g., using the title of each page as potential answers to the questions in the QA pair).

[0035] As another example, the search system can be a web-based system (or associated with a web-based system) that contains crowdsourced questions and answers about web-based services (e.g., for computing, storage, analytics, AI, etc.). These crowdsourced questions and answers may be provided by users of the web-based system. Aggregator component 104 can utilize Structured Query Language (SQL) to create QA pairs using crowdsourced posts that have been tagged by the post's author for a specific web-based service (e.g., how to create a bucket in a specific web-based service). Furthermore, aggregator component 104 can create QA pairs using posts with crowdsourcing acceptance (e.g., a majority of users of the crowdsourcing system mark the post as an accepted answer) or a specific number of likes (e.g., retrieving only posts with at least 5 likes).

[0036] As another example, the search system could be a web-based system (or associated with a web-based system) that contains support tickets from customers accessing web-based services and the resulting solutions (e.g., troubleshooting a customer's inability to access a web-based storage service, server malfunction, etc.). In this search system, customers can create support ticket cases with a title and a detailed description of the problem related to the case. The search system's agent can analyze the case and suggest ways to resolve the problem related to the case, and can also annotate the case (e.g., issues related to debugging a web-based server when it experiences lag). The aggregator component 104 can search for support tickets with annotations and create QA pairs, where resolved cases serve as answers to potential problems.

[0037] As another example, the search system can be a web-based system (or associated with a web-based system) that includes quizzes or tests (e.g., their answers and how customers behave, etc.) to help customers of the web-based service understand how to use it. Aggregator component 104 can use the results of the quizzes or tests to create QA pairs, where some of the quizzes or tests can be answers to questions (e.g., natural language question request quizzes to help users better train on web-based AI systems, etc.).

[0038] As another example, the search system could be a web-based system (or associated with a web-based system) that contains FAQ paragraphs, documents, multimedia, etc., for customers of the web-based service. Aggregator component 104 can create QA pairs based on the FAQ paragraphs (e.g., the answer to the FAQ page is the answer in the QA pair, and the question on the FAQ page is the question in the QA pair).

[0039] As another example, the search system can be a web-based system (or associated with a web-based system) that contains training material (e.g., videos, documents, interactive quizzes, etc.) for web-based services. The aggregator component 104 can create QA pairs, where the training material can be answers to some QA pairs.

[0040] As another example, the search system can be a web-based system (or associated with a web-based system) containing internal questions from employees of the web-based service (e.g., support staff, engineers, etc.) regarding the web-based service (e.g., questions related to employees using the web-based service). Aggregator component 104 can create QA pairs, where the answer to an employee's question can be an answer in the QA pair, and the employee's question can be a question in the QA pair.

[0041] As another example, the search system could be a web-based system (or associated with a web-based system) containing a knowledge graph of customers regarding web-based services (e.g., what services they have, their usage activity, questions customers have previously asked, the types of questions customers have asked and their frequency, their preferences for answers, etc.). Aggregator component 104 could leverage the knowledge graph to create QA pairs, where information about the customer can be the answer in the QA pair (e.g., the customer has 25 buckets in the web-based storage service), and the question from the customer can be the question in the QA pair (e.g., “how many buckets do I have in this network-based storage service?”).

[0042] The aggregator component 104 can also use similarity scores to determine which retrieved paragraphs are relevant (e.g., not out of scope). The retrieved paragraphs can be sent through a dense encoder to obtain their dense embeddings. The aggregator component 104 can generate a score for each paragraph relative to the natural language question (e.g., how relevant the paragraph is to the question). Paragraphs that pass a certain threshold (e.g., greater than 0.5) can be retained, while paragraphs below a certain threshold (e.g., less than or equal to 0.5) can be excluded. Unretained paragraphs can be considered out of scope by the aggregator component 104.

[0043] As described herein, aggregator component 104 can use dense paragraph retrieval (DPR) techniques to determine which paragraphs from search system 124 are relevant to be used as hints to LLM component 106. DPR, often referred to as a technique for open-domain question answering, may aim to retrieve relevant paragraphs from large amounts of unstructured text. Unlike traditional information retrieval techniques that rely on sparse representations, aggregator component 104 can use DPR techniques and dense representations adapted from deep neural networks, where these techniques can be used to encode text paragraphs and questions. Furthermore, aggregator component 104 can use DPR techniques to pre-compute dense vector representations of the text and store them in the search index. For example, aggregator component 104 can use DPR techniques to encode dense representations understood from deep neural networks to encode text paragraphs and questions. Given a question or query, aggregator component 104 can use DPR techniques to retrieve relevant paragraphs from the index based on the similarity between the paragraph representation and the representation of the query or question. Once relevant paragraphs are retrieved, aggregator component 104 can use a downstream model to extract answers from the posed question.

[0044] Aggregator component 104 can create suggestions based on identified relevant paragraphs and natural language questions. Aggregator component 104 can create suggestions that include some of the retrieved paragraphs and QA pairs associated with those paragraphs. Aggregator component 104 can also create suggestions by adding the QA pairs of those paragraphs to the suggestions. For example, aggregator component 104 can formulate suggestions (e.g., and may also obtain QA pairs associated with the document) by retrieving some paragraphs from a document about how to set up a web-based bucket and at least part of a natural language question. Additionally, aggregator component 104 can add questions or some form of question to the suggestions.

[0045] Alternatively, aggregator component 104 may access training dataset 230. Aggregator component 104 may utilize one or more machine learning models (refer to training dataset 230) to determine from which search systems 124 to retrieve paragraphs. Furthermore, aggregator component 104 may utilize one or more machine learning models (refer to training dataset 230) to determine which paragraphs are relevant. These machine learning methods (refer to training dataset 230) may also be performed in addition to the methods and / or techniques described above and herein.

[0046] Figure 2B An embodiment of the architecture of the illustrative validator component 108 is shown. The validator component 108 can be configured to analyze whether the answer provided by the LLM component 106 is incorrectly generated (e.g., no question is answered, the answer is not supported by the training data of the LLM component 106, the answer generated by the LLM component 106 differs from the expected answer, etc.). As described below, the validator component 108 can utilize different components to determine whether the answer from the LLM component 106 is incorrectly generated. The validator component 108 can utilize the text overlap module 234 to determine how much information from the retrieved paragraphs from the aggregator component 104 was used by the LLM component 106. The validator component 108 can utilize the textual natural language reasoning (NLI) module 236 to determine whether the answer generated from the LLM component 106 contradicts the retrieved paragraphs from the aggregator component 104 by using premises and assumptions. The validator component 108 can utilize the relational NLI module 238 to determine whether the answer generated from the LLM component 106 is incorrectly generated from the referenced retrieved paragraphs by using head, tail, and relation triples. Furthermore, the validator component 108 can utilize the member reasoning attack module 240 to determine whether the natural language question is similar to questions previously asked by other clients and whether it is in the training set of the LLM component 106. As shown, the validator component 108 includes a processing unit 220, a network interface 222, a computer-readable media driver 224, and an input / output device interface 226, all of which can communicate with each other via a communication bus. The components of the validator component 108 can be physical hardware components or implemented in a virtualized environment.

[0047] Network interface 222 can provide access to, for example Figure 1 The processing unit 220 is connected to one or more networks or computing systems via network 116. The processing unit 220 can therefore receive information and instructions from other computing systems or services via the network. The processing unit 220 can also communicate with memory 228 and also provide output information for an optional display via input / output device interface 226. In some embodiments, the verifier component 108 may include a comparison... Figure 2B The components shown are those with more (or fewer) components.

[0048] Memory 228 may include computer program instructions executed by processing unit 220 to implement one or more embodiments. Memory 228 typically includes RAM, ROM, or other persistent or non-transitory memory. Memory 228 may store operating system 232, which provides computer program instructions used by processing unit 220 in the general management and operation of verifier component 108. Memory 228 may also include computer program instructions and other information for implementing aspects of this disclosure. For example, in one embodiment, memory 228 includes interface software 230 for communicating with other components or modules of natural language question answering service 102 to verify answers from LLM component 106.

[0049] Additionally, memory 228 may include a text overlap module 234 for determining how much information from the retrieved paragraphs from aggregator component 104 was used by LLM component 106 in the answer to determine whether the answer generated by LLM component 106 was incorrectly generated (e.g., a hallucination). Text overlap module 234 may determine the text overlap between the answer and the retrieved paragraphs using metrics such as recall calculation (e.g., true positives / (true positives + false negatives)) and F1 score (e.g., 2 x (precision * recall / precision + recall)).

[0050] The memory 210 may also include a text NLI module 236 for determining whether an answer generated from the LLM component 106 contradicts a retrieved paragraph from the aggregator component 104 by using premises and assumptions. The text NLI interpretation module 236 may use an NLI model to determine if a contraction exists. The text NLI interpretation module 236 may utilize an NLI model to take two text sequences as input (e.g., assumptions and premises) and determine, given premises, whether the assumption is true (implication), false (contradiction), or undetermined (neutral). The text NLI interpretation module 236 may use an NLI model to determine whether the answer text (hypothesis) from the LLM component 106 contradicts any of the retrieved paragraphs (premises) from the aggregator component 104. If a contradiction exists, the validator component 108 may avoid validating the answer for display to the natural language question client. If a previous answer is contradictory, the natural language question answering service 102 may use the LLM component 106 to generate another answer and use the validator component 108 to validate the new answer.

[0051] Additionally, memory 228 may include a relational NLI module 238 for determining whether an answer generated from LLM component 106 is incorrectly generated (e.g., incomplete, etc.) regarding retrieved paragraphs by using head, tail, and relation triples. Relational NLI module 238 can determine whether an answer is incorrectly generated by extracting (head, tail, relation) triples from entities mentioned in the answer generated by LLM component 106 to entities mentioned by aggregator component 104 in retrieved paragraphs. Relational NLI module 238 can use the extracted triples to create a knowledge graph consisting of entities and relations. Relational NLI module 238 can determine whether the answer contains any triples that contradict the triples extracted from the retrieved paragraphs; if so, relational NLI module 238 can consider the answer to contradict the natural language question associated with those paragraphs. For example, NLI module 238 can identify a head as a web-based service stating that the service it provides is reliable. As another example, NLI module 238 can identify a tail as a web-based service stating that the service it provides is secure. As another example, NLI module 238 can identify relationships based on network-based services and state that the services it provides are scalable.

[0052] The memory 210 may also include a membership inference attack module 240 for determining whether the natural language question is similar to questions previously asked by other customers and whether it is in the training set of the LLM component 106. Generally, membership inference attacks can be a type of privacy attack that attempts to determine whether a specific record or individual is included in the training dataset of a machine learning model. In other words, a potential attacker might try to determine whether a specific data point was used to train the model. A potential attacker could launch a membership inference attack by using a separate dataset similar to the training dataset of the machine learning model. By feeding this dataset into the model and analyzing its output, the attacker might attempt to determine whether a specific record was used to train the model. As used herein, the membership inference attack module 240 can use techniques from membership inference attacks as a way to gain customer trust by showing answers that infer that a sample exists within the training set. The membership inference attack module 240 can use these techniques to determine whether a given sample (e.g., a question or hint) exists in the training distribution. The membership inference attack module 240 can use these techniques to infer the existence of a sample (e.g., a question or hint) in the training set by adding additional complexity, such as "logit scaling," on top of existing loss attacks. The natural language question answering service 102 may provide generated answers only if similar types of questions (or prompts) have previously existed in the training set of the LLM component 106 or if similar samples have been presented to the model during training.

[0053] The validator component 108 can combine the four techniques described above (text overlap module 234, NLI module 236, relational NLI module 238, and membership inference attack module 240) to determine whether to display the answer. For example, the validator component 108 can use the outputs and scores of the four techniques described above to create a threshold score, and if the score is higher than the threshold (e.g., greater than 0.5), the generated answer is provided.

[0054] Furthermore, the verifier component 108 can utilize a detoxification module (not shown) to determine whether the generated answer should be sent to the client computing device 122 that provided the natural language question. The detoxification module of the verifier component 108 can use filters to filter certain types of answers (e.g., answers that provide information about competitors of web-based services, answers containing profanity, answers containing privacy information, etc.). In this example, the natural language question answering service 102 could provide the client with a response such as "I cannot answer this question."

[0055] Additionally, the validator component 108 can access the training dataset 230. The validator component 108 can utilize one or more machine learning models (refer to the training dataset 230) to verify whether the generated answer should be returned to the client computing device 122. These machine learning methods (refer to the training dataset 230) can also be performed in addition to the methods and / or techniques described above and herein.

[0056] Figure 3 This is an exemplary interaction of different components of a natural language question answering service 102 that analyzes natural language questions (or prompts). At (1), a client computing device 122 (e.g., via a user interface (UI) of the client computing device 122) provides a natural language question (or prompt) to the natural language question answering service 102. The question (or prompt) may be input by a user of the client computing device 122, wherein the user is a client of one or more web-based services (e.g., web-based storage, web-based on-demand code execution, etc.). The question (or prompt) may also be input via different methods (e.g., text input of the question or prompt, audio input of the question or prompt, input generated by other generated models, etc.). In addition, input may be received via a graphical user interface, via an API, etc.

[0057] At (2), the Natural Language Question Answering Service 102 utilizes the aggregator component 104 to determine the relevant paragraphs to be retrieved from the search system 124. As described herein, the aggregator component 104 may utilize partial string manipulation and DPR techniques to determine the meaning of the natural language question and may also determine from which web-based services or computing domains the paragraphs should be retrieved. At (3), the aggregator component 104 retrieves paragraphs from the search system 124. Furthermore, the aggregator component 104 may use the retrieved paragraphs to modify and supplement the question to form a prompt. The prompt may include selected paragraphs and QA pairs for the LLM component 106 to provide an answer.

[0058] At (4), aggregator component 104 sends some of the retrieved paragraphs and corresponding QA pairs to user context component 105. At (5), user context component 105 determines the user context associated with the user of client computing device 122 and forwards the QA pairs, retrieved paragraphs, and user context to LLM component 106. It should be noted that (5) is optional, and aggregator component 104 may directly send the retrieved paragraphs and QA pairs to LLM component 106 without a user context. At (6), LLM component 106 receives the prompts and user context and determines the answer to the natural language question. LLM component 106 may receive the prompts and user context from user context component 105 using an API, wherein the prompts may be formatted in some way for sending or transmitting via the API.

[0059] As described herein, LLM component 106 can be a generative AI model using LLM techniques such as RAG. LLM component 106 can be trained on QA pairs from search system 124, giving it a text corpus to understand questions about each of search system 124 (e.g., and each of the network-based systems associated with search system 124). LLM component 106 can be continuously trained by introducing new QA pairs and other information. For example, LLM component 106 can be trained first on data and tasks from the public internet, enabling it to answer questions about general topics (e.g., without needing to know about specific private network-based services). Still in this example, LLM component 106 can then be trained on specific network-based service information (e.g., paragraphs and QA pairs about a particular network-based service, such as a network-based on-demand code execution system). LLM component 106 can also be trained using techniques such as Human Feedback Reinforcement Learning (RLHF) to rank answers based on user feedback.

[0060] LLM component 106 can also be configured to process multiple questions (or prompts) from a user (e.g., a customer of one or more web-based services) from client computing device 122, allowing LLM component 106 to form an evidence pool using previously asked questions (or prompts) and previously provided answers. The evidence pool can be used to provide an answer to the current question. LLM component 106 can store multiple previous questions as a dialogue context. LLM component 106 can use this evidence pool, the current natural language question, and the dialogue context to answer the current question. LLM component 106 can update the evidence pool in different ways. In one example, LLM component 106 can update the evidence pool using any questions and / or answers generated by LLM component 106. In another example, if LLM component 106 determines that the current evidence pool does not contain an answer to the natural language question, LLM component 106 can update the evidence pool. In this case, LLM component 106 can request newer paragraphs to populate the evidence pool (e.g., from aggregator component 104). In another example, LLM component 106 can use an evidence pool to determine whether the current natural language problem should be rewritten based on the information in the evidence pool.

[0061] Additionally, LLM component 106 can be further trained to provide and / or run application programming interface (API) commands in response to natural language questions or prompts (e.g., as command-line interface (CLI) commands). LLM component 106 can receive natural language questions (or prompts) via natural language question answering service 102 that require running commands on a web-based system subscribed to by the user who asked the question (or prompt). LLM component 106 can access the user's credentials (e.g., which services they subscribe to, usage history, customer knowledge graph, etc.) to generate an answer that may include the API command. For example, natural language question answering service 102 can be associated with a web-based storage service. A customer of this web-based storage service can ask LLM component 106 (e.g., a question or prompt) how to create a bucket in the storage service as a natural language question. LLM component 106 can then generate an answer that provides API commands to create the bucket (e.g., LLM component 106 can also run API commands for the customer). As another example, the natural language question answering service 102 can be associated with a web-based service (e.g., analytics, AI, storage, database, on-demand code execution, etc.). Clients of this web-based service can query (e.g., ask questions or provide hints) the health or status of the web-based service to LLM component 106 (e.g., why is the service running slower than usual? Is the service down? Are there any technical issues with the service? etc.). LLM component 106 can then generate an answer that provides API commands to check the health or status of the web-based service (e.g., LLM component 106 can also run API commands for the client). As another example, the natural language question answering service 102 can be associated with a web-based database service. Clients of this web-based database service can query (e.g., ask questions or provide hints) LLM component 106 to read or write (e.g., add or delete rows, etc.) the database associated with the web-based database service. LLM component 106 can then generate an answer that provides API commands to perform read or write operations on the database (e.g., LLM component 106 can also run API commands for the client). As another example, a natural language question-and-answer service 102 can be associated with a web-based on-demand code execution service. A client of this service can inquire (e.g., ask a question or provide a hint) into an LLM component 106 to initiate a function that the service will perform (e.g., a payroll function for paying employee salaries). The LLM component 106 can then generate an answer that provides API commands to execute the function of the service (e.g., the LLM component 106 can also run API commands for the client).As another example, the natural language question-and-answer service 102 can be associated with a web-based artificial intelligence (AI) service. A client of the web-based AI service can ask (e.g., a question or prompt) an LLM component 106 to initiate an AI command that the web-based AI service will execute (e.g., a command to predict when there will be a peak in the purchase of a certain brand of headphones, etc.). The LLM component 106 can then generate an answer that provides an API command (e.g., to execute that API command on the web-based AI service). After determining the type and kind of API command generated on behalf of the client, the LLM component 106 can then send the API command to the client, along with a request or prompt regarding whether the API command should be executed against the web-based service. The client can respond to the LLM component 106 (e.g., or alternatively, the natural language question-and-answer service 102) and instruct the LLM component 106 that the API command should be executed. Upon receiving the request to execute the API command, the LLM component 106 can execute the API command against the web-based service and send a confirmation to the client that the API command was successfully completed.

[0062] Additionally, the LLM component 106 can be further trained to predict answers before being asked a question (or prompted). For example, the LLM component 106 may have a knowledge graph of customers using web-based services and may also have access to actions performed by customers while using the web-based services. In this example, customer movements, clicks, or other computational actions (e.g., via mouse, keyboard, audio listening device, other computational input device, etc.) can be analyzed to determine if any pattern matches that indicate the customer wants to ask a natural language question (or prompt). Therefore, the LLM component 106 can prompt the customer whether they want to ask a certain natural language question and provide an answer before the customer asks the question.

[0063] At (7), LLM component 106 sends the generated answer and retrieved paragraph to validator component 108. At (8), validator component 108 determines whether the answer is incorrectly generated (e.g., a hallucination). As described above, validator component 108 may look for text overlap between the answer and the retrieved paragraph, determine whether there is a contradiction between the answer and the retrieved paragraph, use head / tail / relationship triples to confirm fidelity, use member reasoning attack techniques to confirm whether the question (e.g., or a similar question) is in the dataset, and / or the score of any combination of the four. At (9), if the answer is not a hallucination, validator component 108 sends the answer and retrieved paragraph to attribution component 109.

[0064] At (10), attribution component 109 may provide references to retrieved paragraphs, embedded references to sentences from retrieved paragraphs used in the answer, or provide questions similar to those in natural language questions. For example, attribution component 109 may provide reference links and titles to retrieved paragraphs used by LLM component 106 (e.g., retrieved paragraphs used as the context for generating the answer), which may allow the question submitter to obtain more details about the referenced paragraphs. As another example, attribution component 109 may determine that sentences (e.g., or a phrase) from retrieved paragraphs are used directly (e.g., word-for-word) in the answer generated by LLM component 106 and embed references to the retrieved paragraph (e.g., or a document associated with the retrieved paragraph) into sentences (e.g., or a phrase) used in the answer (e.g., the sentence used in the answer will have a citation to the paragraph or document from which the sentence is taken). As another example, attribution component 109 can provide questions related to the natural language question by referencing a dense index (e.g., a dense index used to create a fixed-dimensional representation of a previous question) and retrieving the k questions closest to the natural language question. References, embedded citations, and similar questions, but not limited to these, can be provided within or after the answer generated by LLM component 106.

[0065] At (11), watermarking component 110 adds a pattern to the answer to make it unique to the natural language question answering service 102 and verifiable to prevent subsequent duplication. In other words, 110 can embed patterns into the generated text of the answer from LLM component 106. These patterns are invisible to humans but can be detected by algorithms from short segments of lexical units (e.g., a group of words or where lexical units are equal to a single word or a single group of characters). Lexical units can be selected before watermarking and can be boosted during watermarking of the generated answer. At (11), watermarking component 110 sends the watermarked answer and the retrieved paragraph to client computing device 122 so that a user of client computing device 122 can view the answer and the retrieved paragraph.

[0066] Figure 4 This is a flowchart illustrating the routine of the natural language question answering service 102 that will be described in answering questions.

[0067] At block 402, the Natural Language Question Answering Service 102 receives a natural language question (or prompt) from one of the client computing devices 122 (e.g., where a user of the device enters a question via a UI). As described herein, the user of the client computing device 122 may be a client of a web-based service associated with the Natural Language Question Answering Service 102. Alternatively, the question may be submitted or entered via an API. The question may also be generated by another generative model (e.g., not LLM component 106), allowing the two models to communicate effectively with each other without human intervention. In this case, the user may be seeking an answer specifically relevant to the web-based service they have subscribed to. For example, the user may ask the Natural Language Question Answering Service 102 how to create a bucket in a web-based storage service. As another example, the user may ask the Natural Language Question Answering Service 102 to create an API call, or to create and run an API call (e.g., the user asks, "please create a bucket for me named 'bucket3' in my network-based storage service").

[0068] At block 404, the natural language question answering service 102 can retrieve relevant paragraphs to answer the question (or prompt) by identifying them through the aggregator component 104. As described above, the aggregator component 104 can utilize partial string matching techniques to identify words or phrases from the question input that may have been misspelled or mistyped by the user (e.g., the user entered "Londin" but meant "London"). Doing so at least determines the meaning of the question.

[0069] At block 406, aggregator component 104 retrieves paragraphs (e.g., documents, links, API calls, multimedia, etc.) related to the answer from search system 124. Aggregator component 104 may retrieve all documents from search system 124, or retrieve certain text (e.g., embedded text) from a document instead of the entire document. As described herein, paragraphs may also be retrieved in the form of QA pairs accompanying the paragraphs, among other forms. Aggregator component 104 generates hints using some of the retrieved paragraphs and questions, where the hints may include portions of the paragraphs and associated QA pairs, as well as some form of the original natural language question (or hint). Aggregator component 104 may use a similarity score to determine whether the retrieved paragraphs and QA pairs exceed a threshold when compared to the natural language question (e.g., if they exceed the threshold, allowing the paragraphs and QA pairs to be returned as hints to LLM component 106). As described above, natural language question answering service 102 may optionally determine the user context through user context component 105, where the user context may also be provided to LLM component 106 so that LLM component 106 generates the answer.

[0070] At block 408, the Natural Language Question Answering Service 102 determines the answer (e.g., in human-readable text) based on prompts (e.g., and in some cases, user context) via the LLM component 106. The LLM component 106 may utilize RAG technology to determine the answer. Furthermore, the LLM component 106 may utilize customer information such as a resource graph including customer subscription information (e.g., current storage usage, current processing usage, CPU cycles, etc.) to determine or refine the answer for the customer (e.g., the LLM component 106 determines that the customer's service subscription will expire next month, therefore including in the answer that the customer should renew their service). The LLM component 106 can also generate API calls based on the question (or run them for the customer) (e.g., the customer wants an API call to create a bucket in a web-based storage service, and the LLM component 106 generates it). Additionally, the LLM component 106 may pre-determine the customer's questions based on the customer's activity (e.g., referencing a knowledge graph and determining that the customer prefers links to other paragraphs rather than answers with long text within the answer itself).

[0071] At block 410, the Natural Language Question Answering Service 102 uses one or more techniques via validator component 108 to verify that the answer is not incorrectly generated (e.g., an illusion). Validator component 108 may utilize text overlap module 234 to determine if there is overlap between the answer and the retrieved paragraph. Validator component 108 may utilize text NLI interpretation module 236 to determine if the answer contradicts the retrieved paragraph. Additionally, validator component 108 may utilize relation NLI module 238 to determine whether the answer generated from LLM component 106 is incorrectly generated for the natural language question by using head, tail, and relation triples. Furthermore, validator component 108 may utilize member reasoning attack module 240 to determine whether the question is similar to previously generated questions and whether it is in the training set of LLM component 106 (e.g., deciding to answer the question based on whether the question or a derivative is already in the training set). Furthermore, if the answer is verified, a watermark can be received via watermark component 110, where a word pattern is generated in the answer that is unreadable to humans, but which the natural language question answering service 102 can use to determine that it generated the answer. In other words, watermark component 110 can obtain the generated answer from LLM component 106 and reorder or rephrase the words to create a pattern that LLM component 106 can determine as the answer it generated, but which is unreadable to humans.

[0072] At block 412, attribution component 109 identifies references, embedded citations, and similar questions. As described above, attribution component 109 can provide reference links and headings to retrieved paragraphs (e.g., retrieved paragraphs used as the context for generating the answer) used by LLM component 106, which can allow the question submitter to obtain more details about the referenced paragraphs. As another example, attribution component 109 can determine that sentences (e.g., or a phrase) from retrieved paragraphs are directly (e.g., word-for-word) used in the answer generated by LLM component 106, and reference the retrieved paragraph (e.g., or a document associated with the retrieved paragraph) embedded in sentences (e.g., or a phrase) used in the answer (e.g., the sentence used in the answer will have a citation referencing the paragraph or document from which the sentence is taken). As another example, attribution component 109 can provide questions related to the natural language question by referencing a dense index (e.g., a dense index used to create a fixed-dimensional representation of the previous question) and retrieving the k questions closest to the natural language question. References, inline citations, and similar questions, but not limited to those, can be provided within or after the answers generated by LLM component 106.

[0073] At block 414, the natural language question answering service 102 determines whether the answer is incorrectly generated (e.g., an illusion). If the answer is incorrectly generated, the routine ends. If not, at block 416, the answer and the retrieved paragraph are sent to the client computing device 122.

[0074] Additional examples Example 1 Example 1: A system for processing queries. The system is configured to receive a question in natural language text from a client computing device and through a user interface (UI) associated with the client computing device; generate supplementary search system results based on the received question through an aggregator of the system; generate hints based on the received question and parts of the supplementary search system results; generate one or more answers based on the generated hints through a large language model (LLM) of the system; determine, through a validator of the system, whether the one or more answers satisfy a threshold representation of whether the one or more answers are incorrectly generated for the question based on a training dataset associated with the LLM; generate one or more of (i) references, (ii) embedded citations, or (ii) questions similar to the question through an attribution component of the system based on the one or more answers and the supplementary search system results; generate watermarked versions of the one or more answers through a watermarking module of the system; and send the watermarked versions of the one or more answers, the supplementary search system results, and the references, embedded citations, or similar questions generated by the attribution component to the UI of the client computing device. The supplementary search system results include an Open Search-based index, which comprises one or more documents from the search system based on the submission of the received question, and a dense index, which identifies portions of one or more documents as relevant to the received question.

[0075] Example 2: In Example 1, the one or more answers comprise at least human-readable text.

[0076] Example 3: In either Example 1 or 2, the watermarked version of the one or more answers is generated by embedding a signal into the human-readable text, such that the embedded signal makes the watermarked version of the one or more answers system-specific.

[0077] Example 4: In any of Examples 1-3, in forming one or more documents by an aggregator, the aggregator is configured to: send a request to the search system to retrieve one or more documents; and receive one or more documents from the search system.

[0078] Example 5: A system for processing queries, configured to: receive a question from a client computing device; generate relevant paragraphs corresponding to search system results based on the received question using the system's aggregator; generate one or more answers based on the question and relevant paragraphs using the system's Large Language Model (LLM); determine, using the system's validator, that the one or more answers are not erroneously generated; and send the one or more answers to the client computing device.

[0079] Example 6: In Example 5, the system as described in claim 5, wherein the relevant paragraphs are also sent to the client computing device.

[0080] Example 7: In either Example 5 or 6, the LLM includes a retrieval augmentation generation (RAG) model.

[0081] Example 8: In any of Examples 5-7, the aggregator retrieves relevant paragraphs from multiple search systems, wherein the multiple search systems are at least one of the following: a search system configured to provide answers related to a web-based storage system; a search system configured to provide answers related to a web-based on-demand code execution system; a search system configured to provide answers related to a web-based database system; or a search system configured to provide answers related to a web-based on-demand computing system.

[0082] Example 9: In any of Examples 5-8, at least one of the multiple search systems is configured to provide answers related to questions from customers of the web-based system.

[0083] Example 10: In any of Examples 5-9, at least one of the plurality of search systems is configured to provide answers related to a FAQ page for web-based systems.

[0084] Example 11: In any of Examples 5-10, the LLM is configured to: store information about the customer of the web-based system, wherein the information at least describes the web-based products subscribed to by the customer and the actions taken by the customer in relation to the web-based products; and train a machine learning model of the LLM using the stored customer information to identify patterns of the customer’s actions in relation to the web-based products.

[0085] Example 12: In any of Examples 5-11, the verifier further includes: a text overlap module; a text natural language inference module; a relational natural language inference module; and a membership inference attack module.

[0086] Example 13: In Example 12, the text overlap module measures the text overlap between relevant paragraphs and one or more answers.

[0087] Example 14: In any of Examples 12-13, the text natural language reasoning module compares the relevant paragraph with one or more answers to ensure that one or more answers do not contradict the relevant paragraph.

[0088] Example 15: In any of Examples 5-14, the relational natural language reasoning module verifies that one or more answers are faithful to a relevant paragraph and is configured to: extract (i) head, (ii) tail, and (iii) relation triples between the text of the relevant paragraph and the text of the one or more answers; and verify that none of the triples of the text of the relevant paragraph contradicts any of the text of the one or more answers.

[0089] Example 16: In any of Examples 5-15, before sending the one or more answers to the client computing device, the system is further configured to generate one or more of (i) references, (ii) embedded citations, or (ii) questions similar to the question based on the one or more answers and the search system results via the system's attribution component.

[0090] Example 17: In any of Examples 5-16, the system is further configured to: determine that at least part of the question relates to generating an Application Programming Interface (API) Command Line Interface (CLI) call to be executed on a web-based service; generate an API CLI command for the question; add the generated API CLI command to one or more answers; send one or more answers with the API CLI command and a prompt about whether the API CLI command should be executed to a client computing device; receive confirmation from the client computing device to run the API CLI command; and execute the API CLI command for the web-based service.

[0091] Example 18: A computer-implemented method for processing a query, the method comprising: receiving a question from a client computing device; forming relevant paragraphs retrieved from search system results based on the received question by an aggregator; generating one or more answers based on the question and relevant paragraphs by a large language model (LLM); determining, by a validator, that the one or more answers are not illusions; and sending the one or more answers and relevant paragraphs to the client computing device.

[0092] Example 19: In Example 18, the relevant paragraph includes at least one of the following: a document, the text of a document, an image of a document, or a video of a document.

[0093] Example 20: In any of Examples 18-19, the one or more answers comprise at least human-readable text.

[0094] Example 21: In any of Examples 18-20, the LLM includes a retrieval enhancement generation (RAG) model.

[0095] Example 22: In any of Examples 18-21, before sending one or more answers to a client computing device, the method further includes: generating a watermarked version of the one or more answers via a watermarking module of the system, wherein the watermarked version of the one or more answers is generated by embedding a signal into human-readable text associated with the one or more answers, such that the embedded signal makes the watermarked version of the one or more answers system-specific.

[0096] Example 23: In any of Examples 18-22, in forming relevant paragraphs by an aggregator, the aggregator is configured to: send a request to the search system to retrieve relevant paragraphs; and receive relevant paragraphs from the search system.

[0097] Example 24: In any of Examples 18-23, before sending the one or more answers to the client computing device, the method further includes: generating one or more of (i) references, (ii) embedded citations, or (ii) questions similar to the question based on the one or more answers and the search system results.

[0098] Example 25: In any of Examples 18-24, the method further includes: determining that at least part of the question relates to generating an application programming interface (API) command line interface (CLI) to be executed on a web-based service; generating an API CLI command for the question; adding the generated API CLI command to one or more answers; sending one or more answers with the API CLI command and a prompt about whether the API CLI command should be executed to a client computing device; receiving confirmation from the client computing device that the API CLI command should be executed; and executing the API CLI command for the web-based service.

[0099] Example 2 Example 26: A system for processing queries, configured to: receive a question in natural language text from a client computing device and via a user interface (UI) associated with the client computing device; generate supplementary search system results based on the received question via an aggregator of the system; generate a hint based on at least a portion of the received question and the supplementary search system results; generate one or more answers based on the generated hint via a large language model (LLM) of the system; generate one or more of the following via an attribution component of the system based on the one or more answers and the supplementary search system results: (i) a reference link to one or more documents as one or more Uniform Resource Identifiers (URIs), (ii) an embedded reference to portions of one or more documents cited verbatim in one or more answers, or (ii) a question similar to the question; and send the one or more answers, the supplementary search system results, and the reference link, embedded reference, or similar question generated by the attribution component to the UI of the client computing device. The supplementary search system results include: an open search-based index, wherein the open search-based index includes one or more documents from the search system based on submissions of the received question; and a dense index, wherein the dense index identifies portions of one or more documents as relevant to the received question.

[0100] Example 27: In Example 26, the one or more answers comprise at least human-readable text.

[0101] Example 28: In either Example 26 or 27, the LLM includes a retrieval enhancement generation (RAG) model.

[0102] Example 29: In any of Examples 26-28, in forming one or more documents by an aggregator, the aggregator is configured to: send a request to the search system to retrieve one or more documents; and receive one or more documents from the search system.

[0103] Example 30: A system for processing queries, the system being configured to: receive a question from a client computing device; generate relevant paragraphs corresponding to search system results based on the received question using an aggregator of the system; generate one or more answers based on the question and relevant paragraphs using a large language model (LLM) of the system; generate one or more of the following based on the one or more answers and the search system results using an attribution component of the system: (i) a reference link to the relevant paragraph as one or more Uniform Resource Identifiers (URIs), (ii) an embedded reference to a portion of the relevant paragraph quoted verbatim in one or more answers, or (ii) a question similar to the question; and send the one or more answers, the relevant paragraphs, and the reference link, embedded reference, or similar question generated by the attribution component to the client computing device.

[0104] Example 31: In Example 30, the relevant paragraphs were also sent to the client computing device.

[0105] Example 32: In either Example 30 or 31, the LLM includes a retrieval enhancement generation (RAG) model.

[0106] Example 33: In any of Examples 30-32, the aggregator retrieves relevant paragraphs from multiple search systems, wherein the multiple search systems are at least one of the following: a search system configured to provide answers related to a web-based storage system; a search system configured to provide answers related to a web-based on-demand code execution system; a search system configured to provide answers related to a web-based database system; or a search system configured to provide answers related to a web-based on-demand computing system.

[0107] Example 34: In any of Examples 30-33, at least one of the plurality of search systems is configured to provide answers related to questions from customers of the web-based system.

[0108] Example 35: In any of Examples 30-34, at least one of the plurality of search systems is configured to provide answers related to a FAQ page for web-based systems.

[0109] Example 36: In any of Examples 30-35, the LLM is configured to: store information about the customer of the web-based system, wherein the information at least describes the web-based products subscribed to by the customer and the actions taken by the customer in relation to the web-based products; and train a machine learning model of the LLM using the stored customer information to identify patterns of the customer’s actions in relation to the web-based products.

[0110] Example 37: In any of Examples 30-36, the one or more answers comprise at least human-readable text.

[0111] Example 38: In any of Examples 30-37, the supplementary search system results include: an open search-based index, wherein the open search-based index includes one or more documents from the search system based on the submission of the received question; and a dense index, wherein the dense index identifies portions of one or more documents as relevant to the received question.

[0112] Example 39: In any of Examples 30-38, the system is further configured to: determine that at least part of the question relates to generating an Application Programming Interface (API) Command Line Interface (CLI) call to be executed on a web-based service; generate an API CLI command for the question; add the generated API CLI command to one or more answers; send one or more answers with the API CLI command and a prompt about whether the API CLI command should be executed to a client computing device; receive confirmation from the client computing device to run the API CLI command; and execute the API CLI command for the web-based service.

[0113] Example 40: A computer-implemented method for processing a query, the method comprising: receiving a question from a client computing device; forming relevant paragraphs retrieved from search system results based on the received question via an aggregator; generating one or more answers based on the question and relevant paragraphs via a large language model (LLM); generating one or more of the following via an attribution component based on the one or more answers and the search system results: (i) a reference link to the relevant paragraph as one or more Uniform Resource Identifiers (URIs), (ii) an embedded reference to a portion of the relevant paragraph quoted verbatim in one or more answers, or (ii) a question similar to the question; and sending the one or more answers, the relevant paragraphs, and the reference link, embedded reference, or similar question generated by the attribution component to the client computing device.

[0114] Example 41: In Example 40, the relevant paragraphs include: document, document text, document image, or document video.

[0115] Example 42: In any of Examples 40-41, the one or more answers comprise at least human-readable text.

[0116] Example 43: In any of Examples 40-42, the LLM includes a retrieval enhancement generation (RAG) model.

[0117] Example 44: In any of Examples 40-43, in forming relevant paragraphs by an aggregator, the aggregator is configured to: send a request to the search system to retrieve relevant paragraphs; and receive relevant paragraphs from the search system.

[0118] Example 45: The method of claim 40, further comprising: determining that the question relates to generating an application programming interface (API) command line interface (CLI) to be executed on a web-based service; generating an API CLI command for the question; adding the generated API CLI command to one or more answers; sending one or more answers with the API CLI command and a prompt about whether the API CLI command should be executed to a client computing device; receiving confirmation from the client computing device to run the API CLI command; and executing the API CLI command for the web-based service.

[0119] Example 3 Example 46: A system for processing queries, configured to: receive a question in natural language text from a client computing device via a user interface (UI) associated with the client computing device; form supplementary search system results based on the received question by an aggregator of the system, wherein the supplementary search system results include one or more documents and portions of one or more documents; generate a hint based on the received question and portions of the supplementary search system results; generate one or more answers based on the generated hints by a large language model (LLM) of the system; and determine whether the one or more answers satisfy a threshold representation for whether the one or more answers are incorrectly generated for the question by a verifier of the system using a text overlap module for measuring the supplementary search system results against a... Text overlap between one or more answers; a text natural language inference module for comparing the supplementary search system results with one or more answers to ensure that one or more answers do not contradict the supplementary search system results; a relational natural language inference module and a membership inference attack module for determining whether the question is (i) similar to questions previously asked by other client computing devices, and (ii) in the training dataset associated with LLM; generating watermarked versions of the one or more answers by embedding an embedding signal into the human-readable text of the one or more answers through the system's watermarking module such that the embedding signal makes the watermarked versions of the one or more answers system-specific; and sending the watermarked versions of the one or more answers and the supplementary search system results to the UI of the client computing device. The relational natural language inference module is configured to: extract (i) head, (ii) tail, and (iii) relation triples between the text of the supplementary search system results and the text of the one or more answers; and verify that none of the triples of the text of the supplementary search system results contradict any of the text of the one or more answers.

[0120] Example 47: In Example 46, the one or more answers comprise at least human-readable text.

[0121] Example 48: In either Example 46 or 47, before sending the UI to the client computing device, the system is further configured to generate one or more of (i) references, (ii) embedded citations, or (ii) questions similar to the question, based on the one or more answers and the results of the supplementary search system via the system's attribution component.

[0122] Example 49: In any of Examples 46-48, in forming one or more documents by an aggregator, the aggregator is configured to: send a request to the search system to retrieve one or more documents; and receive one or more documents from the search system.

[0123] Example 50: A system for processing queries, the system being configured to: receive a question from a client computing device; generate relevant paragraphs corresponding to search system results based on the received question using the system's aggregator; generate one or more answers based on the question and relevant paragraphs using the system's Large Language Model (LLM); determine that the one or more answers are not erroneously generated using the system's validator by utilizing (i) a text overlap module, (ii) a text natural language inference module, (iii) a relational natural language inference module, and (iii) a membership inference attack module; generate a watermarked version of the one or more answers by embedding a signal into the human-readable text of the one or more answers using the system's watermarking module, such that the embedded signal makes the watermarked version of the one or more answers system-specific; and send the watermarked version of the one or more answers to the client computing device.

[0124] Example 51: In Example 50, the relevant paragraphs were also sent to the client computing device.

[0125] Example 52: In either Example 50 or 51, the LLM includes a retrieval augmentation generation (RAG) model.

[0126] Example 53: In any of Examples 50-52, the aggregator retrieves relevant paragraphs from multiple search systems, wherein the multiple search systems are at least one of the following: a search system configured to provide answers related to a web-based storage system; a search system configured to provide answers related to a web-based on-demand code execution system; a search system configured to provide answers related to a web-based database system; or a search system configured to provide answers related to a web-based on-demand computing system.

[0127] Example 54: In any of Examples 50-53, at least one of the plurality of search systems is configured to provide answers related to questions from customers of the web-based system.

[0128] Example 55: In any of Examples 50-54, at least one of the plurality of search systems is configured to provide answers related to a FAQ page for web-based systems.

[0129] Example 56: In any of Examples 50-55, the LLM is configured to: store information about the customer of the web-based system, wherein the information at least describes the web-based products subscribed to by the customer and the actions taken by the customer in relation to the web-based products; and train a machine learning model of the LLM using the stored customer information to identify patterns of the customer’s actions in relation to the web-based products.

[0130] Example 57: In any of Examples 50-56, the text overlap module measures the text overlap between a relevant paragraph and one or more answers.

[0131] Example 58: In any of Examples 50-57, the text natural language reasoning module compares the relevant paragraph with one or more answers to ensure that one or more answers do not contradict the relevant paragraph.

[0132] Example 59: In any of Examples 50-58, the relational natural language reasoning module verifies that one or more answers are faithful to a relevant paragraph and is configured to: extract (i) head, (ii) tail, and (iii) relation triples between the text of the relevant paragraph and the text of the one or more answers; and verify that none of the triples of the text of the relevant paragraph contradicts any of the text of the one or more answers.

[0133] Example 60: A computer-implemented method for processing a query, the method comprising: receiving a question from a client computing device; forming relevant paragraphs retrieved from search system results based on the received question by an aggregator; generating one or more answers based on the question and relevant paragraphs by a large language model (LLM); determining, by a validator, that the one or more answers are not erroneously generated by utilizing (i) a text overlap module, (ii) a text natural language inference module, (iii) a relational natural language inference module, and (iii) a membership inference attack module; generating a watermarked version of the one or more answers by a watermarking module by embedding a signal into the human-readable text of the one or more answers, such that the embedded signal makes the watermarked version of the one or more answers system-specific; and sending the watermarked version of the one or more answers and the relevant paragraphs to the client computing device.

[0134] Example 61: In Example 60, the relevant paragraphs include: a document, the text of a document, an image of a document, or a video of a document.

[0135] Example 62: In any of Examples 60-61, the one or more answers comprise at least human-readable text.

[0136] Example 63: In any of Examples 60-62, in forming relevant paragraphs by an aggregator, the aggregator is configured to: send a request to the search system to retrieve relevant paragraphs; and receive relevant paragraphs from the search system.

[0137] Example 64: In any of Examples 60-63, before sending the one or more answers to the client computing device, the method of Example 15 further includes: generating one or more of (i) references, (ii) embedded citations, or (ii) questions similar to the question based on the one or more answers and the search system results.

[0138] Example 65: In any of Examples 60-64, the method of Example 15 further includes: determining that the problem is related to generating an Application Programming Interface (API) Command Line Interface (CLI) to be executed on a web-based service; generating an API CLI command for the problem; adding the generated API CLI command to one or more answers; sending one or more answers with the API CLI command and a prompt about whether the API CLI command should be executed to a client computing device; receiving confirmation from the client computing device that the API CLI command should be executed; and executing the API CLI command for the web-based service.

[0139] Example 4 Example 66: A system for processing queries, configured to: receive a question in natural language text from a client computing device via a user interface (UI) associated with the client computing device; generate supplementary search system results based on the received question via an aggregator of the system, wherein the supplementary search system results include one or more documents and portions of one or more documents; generate a prompt based on the received question and portions of the supplementary search system results; generate one or more answers based on the generated prompt via a large language model (LLM) of the system; determine that at least a portion of the question relates to generating an application programming interface (API) command line interface (CLI) call to be executed on a web-based service; generate an API CLI command for the question; add the generated API CLI command to one or more answers; send one or more answers with the API CLI command and a prompt regarding whether the API CLI command should be executed to the client computing device; receive confirmation from the client computing device to execute the API CLI command; and execute the API CLI command for the web-based service.

[0140] Example 67: In Example 66, the one or more answers comprise at least human-readable text.

[0141] Example 68: In either Example 66 or 67, the supplementary search system results are also sent to the client computing device.

[0142] Example 69: In any of Examples 66-68, in forming one or more documents by an aggregator, the aggregator is configured to: send a request to the search system to retrieve one or more documents; and receive one or more documents from the search system.

[0143] Example 70: A system for processing queries, the system being configured to: receive a question from a client computing device; generate relevant paragraphs corresponding to search system results based on the received question using an aggregator of the system; generate one or more answers based on the question and relevant paragraphs using a large language model (LLM) of the system; determine that at least part of the question is related to generating an application programming interface (API) command to be executed against a web-based service; generate the API command related to the question based on at least one or more answers; execute the API command on the web-based service to complete the query; and send one or more answers and a notification of API command completion to the client computing device.

[0144] Example 71: In Example 70, the relevant paragraphs were also sent to the client computing device.

[0145] Example 72: In either Example 70 or 71, the LLM includes a retrieval enhancement generation (RAG) model.

[0146] Example 73: In any of Examples 70-72, an aggregator retrieves relevant paragraphs from multiple search systems, wherein the multiple search systems are at least one of the following: a search system configured to provide answers related to a web-based storage system; a search system configured to provide answers related to a web-based on-demand code execution system; a search system configured to provide answers related to a web-based database system; or a search system configured to provide answers related to a web-based on-demand computing system.

[0147] Example 74: In any of Examples 70-73, at least one of the plurality of search systems is configured to provide answers related to questions from customers of the web-based system.

[0148] Example 75: In any of Examples 70-74, at least one of the plurality of search systems is configured to provide answers related to a FAQ page for a web-based system.

[0149] Example 76: In any of Examples 70-75, the LLM is configured to: store information about the customer of the web-based system, wherein the information at least describes the web-based products subscribed to by the customer and the actions taken by the customer in relation to the web-based products; and train a machine learning model of the LLM using the stored customer information to identify patterns of the customer’s actions in relation to the web-based products.

[0150] Example 77: In any of Examples 70-76, before sending the one or more answers to the client computing device, the system is further configured to generate, via the system's attribution component, one or more of (i) references, (ii) embedded citations, or (ii) questions similar to the question, based on the one or more answers and the search system results.

[0151] Example 78: A computer-implemented method for processing a query, the method comprising: receiving a question from a client computing device; forming relevant paragraphs retrieved from search system results based on the received question via an aggregator; generating one or more answers based on the question and relevant paragraphs via a large language model (LLM); determining that at least a portion of the question is related to generating an application programming interface (API) command to be executed against a web-based service; generating the API command related to the question based on at least one or more answers; executing the API command on the web-based service to complete the query; and sending one or more answers and a notification of API command completion to the client computing device.

[0152] Example 79: In Example 78, the relevant paragraphs include: a document, the text of a document, an image of a document, or a video of a document.

[0153] Example 80: In any of Examples 78-79, the relevant paragraph is also sent to the client computing device.

[0154] Example 81: In any of Examples 78-80, the one or more answers comprise at least human-readable text.

[0155] Example 82: In any of Examples 78-81, the LLM includes a retrieval enhancement generation (RAG) model.

[0156] Example 83: In any of Examples 78-82, before sending one or more answers to a client computing device, the method further includes: generating a watermarked version of the one or more answers via a watermarking module, wherein the watermarked version of the one or more answers is generated by embedding a signal into human-readable text associated with the one or more answers, such that the embedded signal makes the watermarked version of the one or more answers system-specific.

[0157] Example 84: In any of Examples 78-83, in forming relevant paragraphs by an aggregator, the aggregator is configured to: send a request to the search system to retrieve relevant paragraphs; and receive relevant paragraphs from the search system.

[0158] Example 85: In any of Examples 78-84, before sending the one or more answers to the client computing device, the method further includes: generating one or more of (i) references, (ii) embedded citations, or (ii) questions similar to the question based on the one or more answers and the relevant paragraphs.

[0159] It should be understood that not all objectives or advantages can be achieved according to any particular embodiment described herein. Therefore, for example, those skilled in the art will recognize that certain embodiments may be configured to operate in a manner that achieves or optimizes one or more advantages as taught herein without necessarily achieving other objectives or advantages as may be taught or implied herein.

[0160] All processes described herein can be fully automated via software code modules (including one or more specific computer-executable instructions executed by a computing system). The computing system may include one or more computers or processors. The code modules may be stored on any type of non-transitory computer-readable medium or other computer storage device. Some or all of the methods described may be embodied in dedicated computer hardware.

[0161] Based on this disclosure, many other variations besides those described herein will become apparent. For example, according to embodiments, certain actions, events, or functions of any of the algorithms described herein may be performed in a different order, or may be added, combined, or completely omitted (e.g., not all described actions or events are necessary for the practice of the algorithm). Furthermore, in some embodiments, operations or events may be executed concurrently rather than sequentially, for example, through multithreading, interrupt handling, or on multiple processors or processor cores or other parallel architectures. Additionally, different tasks or processes may be executed by different machines and / or computing systems that can run together.

[0162] The various illustrative logic blocks and modules described in conjunction with the embodiments disclosed herein may be implemented or executed by a machine, such as a processing unit or processor designed to perform the functions described herein, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The processor may be a microprocessor, but alternatively, it may be a controller, a microcontroller, or a state machine, a combination thereof, etc. The processor may include an electrical circuit system configured to process computer-executable instructions. In another embodiment, the processor includes an FPGA or other programmable device that performs logical operations without processing computer-executable instructions. The processor may also be implemented as a combination of client computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors along with a DSP core, or any other such configuration. While this document describes primarily digital technologies, the processor may also primarily include analog components. The computing environment may include any type of computer system, including but not limited to microprocessor-based computer systems, mainframe computers, digital signal processors, portable client computing devices, computing engines within device controllers or appliances, to name just a few.

[0163] Unless otherwise specifically stated, conditional languages ​​such as “can,” “may,” “may,” or “can” should generally be understood in the context to indicate that certain embodiments include certain features, elements, and / or steps, even though they are not included in other embodiments. Therefore, such conditional languages ​​are generally not intended to imply that features, elements, and / or steps are required in any one or more embodiments in any way, or that one or more embodiments necessarily include logic for determining whether such features, elements, and / or steps are included or will be performed in any particular embodiment, with or without customer input or prompts.

[0164] Unless otherwise specified, antonymous connectives such as the phrase “at least one of X, Y or Z” should be understood in the context as commonly used to indicate that an item, term, etc., can be X, Y or Z or any combination thereof (e.g., X, Y and / or Z). Therefore, such antonymous connectives are generally not intended and should not imply that certain embodiments require the presence of at least one of X, at least one of Y, or at least one of Z respectively.

[0165] Any process description, element, or block in the flowcharts described herein and / or depicted in the accompanying drawings should be understood to potentially represent a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or element in the process. Alternative implementations are included within the scope of the embodiments described herein, wherein, as those skilled in the art will understand, elements or functions may be omitted, performed out of the order shown or discussed, including substantially simultaneous or reverse execution, depending on the functionality involved.

[0166] Unless otherwise expressly stated, articles such as “a” or “an” should generally be understood to include one or more of the described items. Therefore, phrases such as “a device configured to…” are intended to include one or more of the stated devices. Such one or more stated devices may also be configured collectively to perform the stated description. For example, “a processor configured to perform descriptions A, B, and C” may include a first processor configured to perform description A, which works in conjunction with a second processor configured to perform descriptions B and C.

Claims

1. A system for processing queries, the system being configured to: Receive questions in natural language text from the client computing device and through the user interface (UI) associated with the client computing device; The aggregator of the system generates supplementary search system results based on the received questions, wherein the supplementary search system results include: An open search-based index, wherein the open search-based index includes one or more documents from a search system based on the submission of the received question; and Dense indexing, wherein the dense indexing identifies portions of the one or more documents as relevant to the received question; A prompt is generated based on the received question and a portion of the results from the supplementary search system. The system generates one or more answers based on the generated prompts using a large language model (LLM). The system's validator determines, based on the training dataset associated with the LLM, whether one or more answers satisfy a threshold representation generated for the question error. The attribution component of the system generates one or more of the following based on the one or more answers and the results of the supplementary search system: (i) references, (ii) embedded citations, or (ii) questions similar to the question. The system's watermark module generates watermarked versions of one or more answers; and The watermarked version of one or more answers, the supplementary search system results, and references, embedded citations, or similar questions generated by the attribution component are sent to the UI of the client computing device.

2. The system of claim 1, wherein the one or more answers comprise at least human-readable text, and wherein the watermarked version of the one or more answers is generated by embedding a signal into the human-readable text, such that the embedded signal makes the watermarked version of the one or more answers system-specific.

3. The system of claim 1 or claim 2, wherein in forming the one or more documents via the aggregator, the aggregator is configured to: Send a request to the search system to retrieve the one or more documents; and Receive one or more documents from the search system.

4. A system for processing queries, the system being configured to: Problem received from the customer's computing device; The system's aggregator generates relevant paragraphs corresponding to the search system results based on the received questions. The system generates one or more answers based on the question and the relevant paragraphs using a large language model (LLM). The system's validator determines that the one or more answers are not incorrectly generated; and Send one or more answers to the client computing device.

5. The system of claim 4, wherein the relevant paragraph is also sent to the client computing device, wherein the relevant paragraph is retrieved by the aggregator from a plurality of search systems, wherein at least one of the plurality of search systems is configured to provide an answer related to a client's question on a web-based system, and wherein the plurality of search systems is at least one of the following: Configured to provide a search system for answers related to a web-based storage system; Configured to provide a search system for answers related to a web-based, on-demand code execution system; Configured to provide search systems that offer answers related to web-based database systems; or Configured to provide search systems that provide answers related to web-based on-demand computing systems.

6. The system of claim 5, wherein at least one of the plurality of search systems is configured to provide answers related to a frequently asked questions (FAQ) page for a web-based system.

7. The system of any one of claims 4-6, wherein the LLM is configured as follows: The system stores information about the customer of the web-based system, wherein the information at least describes the web-based products subscribed to by the customer and the actions taken by the customer regarding the web-based products; and The machine learning model of the LLM is trained using stored customer information to identify patterns of customer actions related to the web-based product.

8. The system of any one of claims 4-7, wherein the verifier further comprises: Text overlap module; Textual Natural Language Inference Module; Relational Natural Language Reasoning Module; as well as Member inference attack module.

9. The system of claim 8, wherein the text overlap module measures the text overlap between the relevant paragraph and the one or more answers, wherein the text natural language inference module compares the relevant paragraph with the one or more answers to ensure that the one or more answers do not contradict the relevant paragraph, and wherein the relational natural language inference module verifies that the one or more answers are faithful to the relevant paragraph and is configured to: Extract the (i) beginning, (ii) end, and (iii) relation triples between the text of the relevant paragraph and the text of one or more answers; and Verify that none of the triples in the text of the relevant paragraphs contradict any of the texts of the one or more answers.

10. The system of claim 4, wherein before sending the one or more answers to the client computing device, the system is further configured to: generate, via the system's attribution component, one or more of (i) references, (ii) embedded citations, or (ii) questions similar to the question, based on the one or more answers and the search system results.

11. The system of claim 4, wherein the system is further configured to: It was determined that at least part of the problem was related to generating application programming interface (API) command-line interface (CLI) calls to be executed on a web-based service; Generate API CLI commands related to the question; Add the generated API CLI command to one or more of the answers; Send the one or more answers with the API CLI command and a prompt about whether the API CLI command should be executed to the client computing device; Receive confirmation from the client computing device that the API CLI command has been executed; as well as Execute the API CLI command for the network-based service.

12. A computer-implemented method for processing queries, the method comprising: Problem received from the customer's computing device; The aggregator generates relevant paragraphs from the search system results based on the received questions. One or more answers are generated based on the question and the relevant paragraphs using a large language model (LLM); The verifier determines that one or more of the answers are not hallucinations; as well as Send the one or more answers and the relevant paragraphs to the client computing device.

13. The computer-implemented method of claim 12, wherein the relevant paragraph comprises at least one of: a document, text of the document, an image of the document, or a video of the document, and wherein the aggregator is configured to: Send a request to the search system to retrieve the relevant paragraphs; and Receive the relevant paragraphs from the search system.

14. The computer-implemented method of claim 12 or claim 13, further comprising, before sending the one or more answers to the client computing device: The system's watermark module generates watermarked versions of the one or more answers, wherein the watermarked versions of the one or more answers are generated by embedding a signal into human-readable text associated with the one or more answers, such that the embedded signal makes the watermarked versions of the one or more answers system-specific, or Based on the one or more answers and the search system results, generate one or more of (i) references, (ii) embedded citations, or (ii) questions similar to the question.

15. The computer-implemented method according to any one of claims 12-14, further comprising: It has been determined that at least part of the problem relates to generating an application programming interface (API) command-line interface (CLI) to be executed on a web-based service; Generate API CLI commands related to the question; Add the generated API CLI command to one or more of the answers; Send the one or more answers with the API CLI command and a prompt about whether the API CLI command should be executed to the client computing device; Receive confirmation from the client computing device that the API CLI command has been executed; as well as Execute the API CLI command for the network-based service.

Images