Systems and methods for knowledge answering engine

Abstract

A method may display, on a user device, a prompt to a user to query an enterprise system; receive, from the user device, an input from the user, wherein the input is a query to the enterprise system; determine, using a first classifier, a context of the query; determine, using a second classifier, a complexity level of the query; determine, using a third classifier, an SQL query and / or a code snippet based on the context of the query and the complexity level of the query; generate at least one of a resultant table and a resultant output based on the SQL query and / or the code snippet and executing the SQL query and / or the code snippet on a database of the enterprise system; and generate and output, on the user device, an electronic summary relating to the SQL query and / or the code snippet and / or based on the resultant table and / or output.

Description

SYSTEMS AND METHODS FOR KNOWLEDGE ANSWERING ENGINECROSS-REFERENCE TO RELATED APPLICATION(S)

[0001] This patent application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 637,579, filed on April 23, 2024, the entirety of which is incorporated herein by reference.TECHNICAL FIELD

[0002] Various embodiments of the present disclosure relate generally to machine learning techniques for generating summarizations and insights and, more particularly, to systems and methods for extracting and processing user inputs as they relate to business analytics.BACKGROUND

[0003] Generative artificial intelligence (Al) applications that exist today focus on the task of using text to generate an image, video, or audio (or a combination of video and audio). This is done by using generative Al techniques to learn the mapping from one modality to the other. A use of this technology is also found in a conversational aspect, where refinements on an initial description can occur to improve the output.

[0004] Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted to be prior art, or suggestions of the prior art, by inclusion in this section.SUMMARY OF THE DISCLOSURE

[0005] In some aspects, the techniques described herein relate to a method including: displaying, on a user device, a prompt to a user to query an enterprise system; receiving, from the user device, an input from the user, wherein the input is a query to the enterprise system; determining, using a first classifier, a context of the received query; determining, using a second classifier, a complexity level of the received query; determining, using a third classifier, an SQL query and / or a predefined code snippet based on the determined context of the received query and the complexity level of the received query, wherein the pre-defined code snippet includes an API call from at least one data model, an application output, or an agent; generating a resultant table based on the determined SQL query and / or pre-defined code snippet and executing the SQL query and / or pre-defined code snippet on a database of the enterprise system; generating a resultant output based on thedetermined SQL query and / or pre-defined code snippet and executing the SQL query and / or pre-defined code snippet on a database of the enterprise system, wherein the resultant output includes at least one of a graph, a chart, or a plot; and generating and outputting, on the user device, an electronic summary relating to the SQL query and / or pre-defined code snippet of the enterprise system based on the resultant table and / or the resultant output, wherein the electronic summary further includes displaying at least one of the SQL query or the pre-defined code snippet.

[0006] In some aspects, the techniques described herein relate to a method including: displaying, on a user device, a prompt to a user to query an enterprise system; receiving, from the user device, an input from the user, wherein the input is a query to the enterprise system; determining, using a first classifier, a context of the received query; determining, using a second classifier, a complexity level of the received query; determining, using a third classifier, an SQL query and / or a predefined code snippet based on the determined context of the received query and the complexity level of the received query; generating at least one of a resultant table and a resultant output based on the determined SQL query and / or code snippet (which may be pre-defined) and executing the SQL query and / or pre-defined code snippet on a database of the enterprise system; wherein the resultant output includes at least one of a graph, a chart, or a plot; and generating and outputting, on the user device, an electronic summary relating to the SQL query and / or pre-defined code snippet of the enterprise system and / or based on the resultant table and / or the resultant output , wherein the electronic summary preferably further includes displaying at least one of the SQL query or the pre-defined code snippet. The method allows entry and processing of a vast array of queries by using context and complexity to direct the queries appropriately. Moreover, the results are provided in user-friendly graphical form, together with an electronically-produced summary which can aid user acceptance and trust of the method. Also, use of the chain of classifiers reduces the processing load for determination of the query results.

[0007] In some aspects, the techniques described herein relate to a method, wherein the (pre-defined) code snippet includes an API call from at least one data model, an application output, or an agent. Of course, other modalities can be used, but a simple API allows an efficient link to the data model / application / agent.

[0008] In some aspects, the techniques described herein relate to a method, wherein the first classifier includes a first LLM that classifies the query based on a validity of the context using database schema and business context.

[0009] In some aspects, the techniques described herein relate to a method, wherein the first classifier rejects the query as out of scope when it is classified as having an invalid context and does not pass to the next layer. This may be a security advantage as well as preserving processing resources.

[0010] In some aspects, the techniques described herein relate to a method, further comprising context enrichment to enhance the context (and information in the general sense) surrounding the user query, the context enrichment preferably including use of database metadata and database dictionaries and / or clarifying questions posed to the user. This intermediate step can make the query more specific and accurate.

[0011] In some aspects, the techniques described herein relate to a method, wherein the determination of the context of the query includes used of personabased and / or custom access data sources for individuals within the enterprise, as well as one or more other data sources. This can personalize the system for better results.

[0012] In some aspects, the techniques described herein relate to a method, wherein information from the persona-based data source and the one or more data sources is passed through an orchestrator agent to determine where each piece of information is to be placed, wherein a data source agent determines the original location of each data item and categorizes it accordingly, the data items then being sent to an SQL code generator for processing.

[0013] In some aspects, the techniques described herein relate to a method, wherein determining the complexity level of the received query (preferably using the second classifier) further includes: determining at least one of a level 1 , a level 2, a level 3, or a level 4. The complexity may increase from level 1 to level 4.

[0014] In some aspects, the techniques described herein relate to a method, wherein the third classifier includes a text-to-SQL LLM.

[0015] In some aspects, the techniques described herein relate to a method, wherein the level 1 complexity is answered using near zero context (for example using only database schema information), wherein the level 2 complexity is answered using (the database) schema (information) and the determined context,wherein the level 3 complexity uses a chain-of-thought framework (for example to break down the query into sub-queries), and wherein level 4 complexity is strategic and planning (this may be in the sense that some context needs to be estimated future context) .

[0016] In some aspects, the techniques described herein relate to a method, further comprising requesting additional information from the user, for example if the query has a complexity level above a complexity threshold and / or to avoid hallucinations and / or to satisfy pre-determined business-rules.

[0017] In some aspects, the techniques described herein relate to a method, wherein a level 4 complexity query is handled in a separate tool from level 1 , level 2, and level 3 queries, for example by calling a simulator or forecasting tool.

[0018] In some aspects, the techniques described herein relate to a method, further comprising selecting a timeframe for use with the query using a timeframe selector which determines a default timeframe or custom timeframe to be used. For example, the custom time frame may be specified by the user, whether in the query or separately. A default timeframe may be a system-wide default or determined in any other suitable way.

[0019] In some aspects, the techniques described herein relate to a method, wherein determining, using the third classifier, the SQL query includes using at least one or natural language programming (NLP), natural language processing, or Large Language Model.

[0020] In some aspects, the techniques described herein relate to a method, wherein any SQL query error message is fed back through the third classifier in a next iteration along with the query, database schema, and the determined context of the query. This feedback allows system learning from the error message.

[0021] In some aspects, the techniques described herein relate to a method, further comprising displaying the resultant output on the user device and preferably also displaying any assumptions used in arriving at the resultant output or in arriving at a next resultant output. Display of assumptions can aid user comprehension.

[0022] In some aspects, the techniques described herein relate to a method, further including displaying an additional prompt to the user after outputting, on the user device, the resultant output and / or the (electronic) summary (for example based on the resultant table and / or resultant output). The prompt may ask for feedback, request a new query, or suggest a specific follow-up query.

[0023] In some aspects, the techniques described herein relate to a method, further comprising running an improved query in response to the user’s response to the additional prompt.

[0024] In some aspects, the techniques described herein relate to a method, wherein the resultant output is subject to evaluation, preferably according to at least one of a check for the information contained in the output against what is stored in the cache and a check of user persona for outputting information or visualizations in a particular format.

[0025] In some aspects, the techniques described herein relate to a system including: a memory configured to store instructions; and one or more processors configured to execute the instructions to perform operations including: displaying, on a user device, a prompt to a user to query an enterprise system; receiving, from the user device, an input from the user, wherein the input is a query to the enterprise system; determining, using a first classifier, a context of the received query; determining, using a second classifier, a complexity level of the received query; determining, using a third classifier, an SQL query and / or a pre-defined code snippet based on the determined context of the received query and the complexity level of the received query, wherein the pre-defined code snippet includes an API call from at least one data model, an application output, or an agent; generating a resultant table based on the determined SQL query and / or pre-defined code snippet and executing the SQL query and / or pre-1defined code snippet on a database of the enterprise system; generating a resultant output based on the determined SQL query and / or pre-defined code snippet and executing the SQL query and / or pre-defined code snippet on a database of the enterprise system, wherein the resultant output includes at least one of a graph, a chart, or a plot; and generating and outputting, on the user device, an electronic summary relating to the SQL query and / or pre-defined code snippet of the enterprise system based on the resultant table and / or the resultant output, wherein the electronic summary further includes displaying at least one of the SQL query or the pre-define code snippet.

[0026] In some aspects, the techniques described herein relate to a system including: a memory configured to store instructions; and one or more processors configured to execute the instructions to perform operations according to any of the preceding method aspects.

[0027] In some aspects, the techniques described herein relate to a system, wherein the first classifier includes a first LLM that classifies the query based on a validity of the context using database schema and business context.

[0028] In some aspects, the techniques described herein relate to a system, wherein determining the complexity level of the received query further includes: at least one of a level 1 , a level 2, a level 3, or a level 4.

[0029] In some aspects, the techniques described herein relate to a system, wherein the third classifier includes a text-to-SQL LLM.

[0030] In some aspects, the techniques described herein relate to a system, wherein the level 1 complexity is answered using near zero content, wherein the level 2 complexity is answered using schema and the determined context, wherein the level 3 complexity uses a chain-of-thought framework, and wherein level 4 complexity is strategic and planning.

[0031] In some aspects, the techniques described herein relate to a system, wherein determining, using the third classifier, the SQL query includes using at least one of natural language programming (NLP), natural language processing, or Large Language Model.

[0032] In some aspects, the techniques described herein relate to a system, further includes displaying an additional prompt to the user after outputting, on the user device, the summary based on the resultant table and / or resultant output.

[0033] In some aspects, the techniques described herein relate to a non- transitory computer-readable medium storing instructions that, when executed by one or more processor, cause the one or more processors to perform operations including: displaying, on a user device, a prompt to a user to query an enterprise system; receiving, from the user device, an input from the user, wherein the input is a query to the enterprise system; determining, using a first classifier, a context of the received query; determining, using a second classifier, a complexity level of the received query; determining, using a third classifier, an SQL query and / or a predefined code snippet based on the determined context of the received query and the complexity level of the received query, wherein the pre-defined code snippet includes an API call from at least one data model, an application output, or an agent; generating a resultant table based on the determined SQL query and / or pre-defined code snippet and executing the SQL query and / or pre-defined code snippet on a database of the enterprise system; generating a resultant output based on thedetermined SQL query and / or pre-defined code snippet and executing the SQL query and / or pre-defined code snippet on a database of the enterprise system, wherein the resultant output includes at least one of a graph, a chart, or a plot; and generating and outputting, on the user device, an electronic summary relating to the SQL query and / or pre-defined code snippet of the enterprise system based on the resultant table and / or the resultant output, wherein the electronic summary further includes displaying at least one of the SQL query or the pre-define code snippet.

[0034] In some aspects, the techniques described herein relate to a computer program, or a non-transitory computer-readable medium storing instructions that, when executed by one or more processor, cause the one or more processors to perform operations comprising the steps of any of the preceding method aspects.

[0035] In some aspects, the techniques described herein relate to a non- transitory computer-readable medium, wherein the first classifier includes a first LLM that classifies the query based on a validity of the context using database schema and business context.

[0036] In some aspects, the techniques described herein relate to a non- transitory computer-readable medium, wherein determining the complexity level of the received query further includes: at least one of a level 1 , a level 2, a level 3, or a level 4.

[0037] In some aspects, the techniques described herein relate to a non- transitory computer-readable medium, wherein the third classifier includes a text-to- SQL LLM.

[0038] In some aspects, the techniques described herein relate to a non- transitory computer-readable medium, wherein the level 1 complexity is answered using near zero content, wherein the level 2 complexity is answered using schema and the determined context, wherein the level 3 complexity uses a chain-of-thought framework, and wherein level 4 complexity is strategic and planning.

[0039] In some aspects, the techniques described herein relate to a non- transitory computer-readable medium, wherein determining, using the third classifier, the SQL query includes using at least one of natural language programming (NLP), natural language processing, or Large Language Model.

[0040] In some aspects, the techniques described herein relate to a non- transitory computer-readable medium, further includes displaying an additionalprompt to the user after outputting, on the user device, the summary based on the resultant table and / or resultant output.

[0041] In some aspects, the techniques described herein relate to a method including: displaying, on a user device, a prompt to a user to query an enterprise system; receiving, from the user device, an input from the user, wherein the input is a query to the enterprise system; determining, using a first classifier, a context of the query; determining, using a second classifier, a complexity level of the query; determining, using a third classifier, an SQL query and / or a code snippet based on the context of the query and the complexity level of the query; generating at least one of a resultant table and a resultant output based on the SQL query and / or the code snippet and executing the SQL query and / or the code snippet on a database of the enterprise system; and generating and outputting, on the user device, an electronic summary relating to the SQL query and / or the code snippet of the enterprise system and / or based on the resultant table and / or the resultant output, wherein the electronic summary preferably further includes displaying at least one of the SQL query or the code snippet.

[0042] In some aspects, the techniques described herein relate to a method, wherein the code snippet includes an API call from at least one data model, an application output, or an agent.

[0043] In some aspects, the techniques described herein relate to a method, wherein the first classifier includes a first LLM that classifies the query based on a validity of the context using database schema and business context.

[0044] In some aspects, the techniques described herein relate to a method, wherein the first classifier rejects the query as out of scope when it is classified as having an invalid context and does not pass to a next layer.

[0045] In some aspects, the techniques described herein relate to a method, further including context enrichment to enhance the context and information surrounding the user query, the context enrichment preferably including use of database metadata and database dictionaries and / or clarifying questions posed to the user.

[0046] In some aspects, the techniques described herein relate to a method, wherein determining the context of the query includes use of a persona-based data source and / or a custom access data source for individuals within the enterprise system, as well as one or more other data sources.

[0047] In some aspects, the techniques described herein relate to a method, wherein information from the persona-based data source and the one or more other data sources is passed through an orchestrator agent to determine where each piece of information is to be placed, wherein a data source agent determines an original location of each data item and categorizes it accordingly, each data item then being sent to an SQL code generator for processing.

[0048] In some aspects, the techniques described herein relate to a method, wherein determining the complexity level of the query using the second classifier further includes: determining at least one of a level 1 complexity, a level 2 complexity, a level 3 complexity, or a level 4 complexity.

[0049] In some aspects, the techniques described herein relate to a method, wherein the level 1 complexity is answered using only database schema information, wherein the level 2 complexity is answered using the database schema information and the context of the query, wherein the level 3 complexity is answered using a chain-of-thought framework to break down the query into sub-queries, and wherein the level 4 complexity is answered using one or more autonomous agents.

[0050] In some aspects, the techniques described herein relate to a method, further including requesting additional information from the user if the query has a complexity level above a complexity threshold and / or to avoid hallucinations and / or to satisfy pre-determined business-rules.

[0051] In some aspects, the techniques described herein relate to a method, wherein a level 4 complexity query is broken down into queries of lower complexity level and / or handled in a separate tool from level 1 , level 2, and level 3 queries, for example by calling a simulator or forecasting tool.

[0052] In some aspects, the techniques described herein relate to a method, further including selecting a timeframe for use with the query using a timeframe selector which determines a default timeframe or custom timeframe to be used.

[0053] In some aspects, the techniques described herein relate to a method, wherein the third classifier includes a text-to-SQL LLM.

[0054] In some aspects, the techniques described herein relate to a method, wherein determining, using the third classifier, the SQL query includes using at least one or natural language programming (NLP), natural language processing, or Large Language Model.

[0055] In some aspects, the techniques described herein relate to a method, wherein any SQL query error message is fed back through the third classifier in a next iteration along with the query, database schema, and the context of the query.

[0056] In some aspects, the techniques described herein relate to a method, further including displaying the resultant output on the user device and preferably also displaying any assumptions used in arriving at the resultant output or in arriving at a next resultant output.

[0057] In some aspects, the techniques described herein relate to a method, further including displaying an additional prompt to the user after outputting, on the user device, the resultant output and / or the electronic summary.

[0058] In some aspects, the techniques described herein relate to a method, further including running an improved query in response to a user response to the additional prompt.

[0059] In some aspects, the techniques described herein relate to a method, wherein the resultant output is subject to evaluation, preferably according to at least one of a check for information contained in the resultant output against what is stored in cache and a check of user persona for outputting information or visualizations in a particular format.

[0060] In some aspects, the techniques described herein relate to a system including: a memory configured to store instructions; and one or more processors configured to execute the instructions to perform operations according to any of the preceding methods.

[0061] In some aspects, the techniques described herein relate to a computer program, or a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations including the steps.

[0062] Additional objects and advantages of the disclosed embodiments will be set forth in part in the description that follows, and in part will be apparent from the description, or may be learned by practice of the disclosed embodiments. The objects and advantages of the disclosed embodiments will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims. As will be apparent from the embodiments below, an advantage to the disclosed systems and methods is that multiple parties may fully utilize their data without allowing others to have direct access to raw data. The disclosed systemsand methods discussed below may allow advertisers to understand users' online behaviors through the indirect use of raw data and may maintain privacy of the users and the data.

[0063] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments, as claimed.BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and together with the description, serve to explain the principles of the disclosed embodiments.

[0065] FIG. 1 A depicts an exemplary flowchart showing a method according to one or more embodiments.

[0066] FIG. 1 B depicts an exemplary networked computing environment for executing a machine-learning model, according to one or more embodiments.

[0067] FIGs. 2A-2C depict exemplary flowcharts using a multi-step chained LLM task flow, according to one or more embodiments.

[0068] FIG. 3 depicts an exemplary flowchart using both Gen Al and non-Gen Al components, according to one or more embodiments.

[0069] FIGs. 4A-4D depict exemplary flowcharts and sample user prompts for a knowledge answering engine, according to one or more embodiments.

[0070] FIGs. 5A-5C depict an exemplary flowchart and sample user interface of a knowledge answering engine, according to one or more embodiments.

[0071] FIGs. 6A-6D depict exemplary flowcharts including user interaction with a knowledge answering engine, according to one or more embodiments.

[0072] FIGs. 7A-7B depict an exemplary flowchart architecture of a knowledge answering engine, according to one or more embodiments.

[0073] FIG. 8 depicts an exemplary architecture diagram of a knowledge answering engine, according to one or more embodiments.

[0074] FIG. 9 depicts a further exemplary architecture diagram of a knowledge answering engine, according to one or more embodiments.

[0075] FIG. 10 illustrates an implementation of a computer system that executes techniques presented herein.DETAILED DESCRIPTION OF EMBODIMENTS

[0076] Various embodiments of the present disclosure relate generally to machine learning techniques for generating summarizations and insights and, more particularly, to systems and methods for extracting and processing user inputs as they relate to business analytics.

[0077] A common challenge in the field of enterprise data analytics for business intelligence includes enabling non-technical business users to perform data analysis, generate understandings, and retrieve knowledge across massive enterprise data sources. Even with the proliferation of Business Intelligence (Bl) tools, these platforms still often demand substantial training, coding skill, or analytic expertise, which business associates often lack. Thus, the process for extracting actionable understandings from data sources is labor-intensive and time-consuming. Moreover, the process frequently falls short of meeting the subtle and granular individual needs of users. Many data formats, comprehension types, and user intents make it harder for data teams to give custom analytic solutions fast, thus causing a scalability issue.

[0078] The disclosure described herein addresses these shortcomings with a new architecture, built around a chained series of Large Language Model (LLM) tasks designed for functioning as an automated task-flow agent. Because the system is anchored on one single connected data foundation layer using a vector database of enterprise data, the system allows all users to interact via one front-end chat interface, thereby enabling natural language querying. The core of the architecture is a four-step LLM-powered pipeline: intent and domain understanding via two LLM- based classifiers, a text-to-SQL (or pre-generated code / API) generation step, and a query result summarization phase. This approach mitigates issues like hallucination by leveraging the strengths of LLMs in classification tasks, adds guardrails to prevent misinformation, and ensures transparency by exposing underlying SQL queries. Integrating this architecture with user feedback mechanisms, contextual prompts, and ambiguity-resolution features further enhances its adaptability and trustworthiness, significantly lowering the barrier for business users while streamlining development efforts for analytics teams.

[0079] The terminology used below may be interpreted in its broadest reasonable manner, even though it is being used in conjunction with a detailed description of certain specific examples of the present disclosure. Indeed, certainterms may even be emphasized below; however, any terminology intended to be interpreted in any restricted manner will be overtly and specifically defined as such in this Detailed Description section.

[0080] FIG. 1 A depicts a method according to invention embodiments. The method starts at S10 by displaying, on a user device, a prompt to a user to query an enterprise system. In S20, the computing system carrying out the method receives, from the user device (for example over a network connection), an input from the user, wherein the input is a query to the enterprise system. There may be three layers of classifiers (e.g., a first classifier, a second classifier, and a third classifier) that are used by the subsequent steps. Step S30 determines, using a first classifier, a context of the received query. Step S40 determines, using a second classifier, a complexity level of the received query. In step S50, a third classifier determines an SQL query and / or a snippet based on the determined context of the received query and the complexity level of the received query. While the first two classifiers may work in parallel, the third classifier must be after the first two classifiers as the third classifier uses the output of the first two classifiers. The code snippet may include an API call from at least one data model, an application output or an agent, so the third classifier can produce any of these for further use in the method to answer the query.

[0081] In the following step S60, a resultant table is generated. In step S70 a resultant output is generated. Both the resultant table and the output may be based on the SQL query and / or code snippet. The output may include one or more of a graph, a chart, and / or a plot, depending on the type of query.

[0082] Finally, in S80, the method generates, and outputs on the user device (such as a terminal device), an electronic summary relating to the SQL query (and / or code snippet of the enterprise system). Alternatively or additionally, an electronic summary may be based on the resultant table and / or the resultant output. The electronic summary may further include displaying at least one of the SQL query or the pre-defined code snippet.

[0083] As depicted in FIG. 1 B, network computing environment 100, which may carry out the above method and other methods of embodiments described herein, may generally include an electronic network 110 in communication with a computing system 102, a user device 112, and a data store 114. In various embodiments, the electronic network 110 may be a wide area network (“WAN”), a local area network (“LAN”), personal area network (“PAN”), or the like. In someembodiments, electronic network 110 includes the Internet, and information and data provided between various systems occurs online. “Online” may mean connecting to or accessing source data or information from a location remote from other devices or networks coupled to the Internet. Alternatively, “online” may refer to connecting or accessing an electronic network (wired or wireless) via a mobile communications network or device. The Internet is a worldwide system of computer networks — a network of networks in which a party at one computer or other device connected to the network can obtain information from any other computer and communicate with parties of other computers or devices. The most widely used part of the Internet is the World Wide Web (often-abbreviated “WWW’ or called “the Web”). A “website page” generally encompasses a location, data store, or the like that is, for example, hosted and / or operated by a computer system so as to be accessible online, and that may include data configured to cause a program such as a web browser to perform operations such as send, receive, or process data, generate a visual display and / or an interactive interface, or the like.

[0084] As shown in FIG. 1 B, in one embodiment, computing system 102 may include a data stream module 104, a display generation module 106, and a transmission module 108. Although depicted as separate components in FIG. 1 B, it should be understood that any component or portion of a component in the network computing environment 100 may, in some embodiments, be integrated with or incorporated into one or more other components. In another example, the computing system 102 may be integrated in a data storage system. The data storage system may be configured to communicate and / or receive / send data across electronic network 110 to other components of network computing environment 100. In some embodiments, operations or aspects of one or more of the components discussed above may be distributed amongst one or more other components. Any suitable arrangement and / or integration of the various systems and devices of the network computing environment 100 may be used.

[0085] FIGs. 2A-2B depict exemplary flowcharts using a multi-step chained LLM task flow, according to one or more embodiments. FIG. 2A depicts a flowchart (e.g., a multi-step architecture) 200 using chained large language models (LLM). The flowchart 200 may utilize Text-to-SQL conversion (e.g., a natural language question to be converted into a SQL query which is then passed to the database to get relevant insights in the form of a table and added visualization features generated).For example, Azure OpenAI’s LLM GPT has been used in realizing the Text-to-SQL conversion given a user query, with database schema and customized business context as inputs. Enhancements in the results are obtained upon iterative prompt engineering at each step. The solution approach has multiple layers, each for a specific task, as explained in more detail herein.

[0086] The flowchart 200 starts with a user query 201 . The user can input a query into a user interface prompt via input (e.g., text, touch, voice, etc.) to query an enterprise system. The system (e.g., enterprise system) may then use a first classifier LLM 203 that classifies a query based on the validity of context and domain for the current scope. The LLM 203 as a classifier may be employed here to classify a query with database schema and business context as inputs. To perform this, the question may be passed as an input as part of a prompt to the LLM 203, which may then determine if the question asked is answerable based on the current data. Extensive prompt engineering may be carried out to customize the LLM 203 response which may then be used to decide whether a query may move to the next layer. Inclusion of this classifier layer may improve latency of the solution.

[0087] The system may determine the context choice 205 of the user query is out-of-context or out-of-domain using the classifier. Out-of-context queries may carry no relevance to the business use case. For example, an out-of-context query may be “When was Mars founded?” Out-of-domain queries may include information outside of the domain of the data provided. For example, an out-of-domain query may be “How are our sales compared to competitors?” If the system determines “YES” (e.g., the query is out-of-context or out-of-domain), the system may proceed to send the information as out-of-scope 207.

[0088] Upon passing the first classifier 203 (e.g., context / domain classifier), if the system 200 determines “NO”, the user query may be required to be correctly placed in one of the four complexities of queries (L1 , L2, L3 or L4) as described below. Thus, the second classifier may classify the complexity level of the query. A second classifier LLM 209 may be utilized to determine the complexity 211 of the query. This step is necessary because each complexity 211 of each query requires either different kinds of prompting or different solution approaches to obtain desired results. The most obvious example is query complexity L4 versus the rest of the query complexities L1-L3. The L4 query would require tools corresponding to various models such as simulator, forecasting etc., whereas the rest of the complexities onlyrequire the nominal Text-to-SQL tool 213 utilized by a LLM 215, making the solution approaches different between the two. L1 complexity (e.g., level 1 complexity) relates to user queries that can be answered with near zero context using only the database schema information provided. For example, these user queries may include user queries relating to simple aggregations, trends over time of pre-existing field, absolute vs. relative timeframes, and financial metrics (e.g., sales, etc.). L2 complexity (e.g., level 2 complexity) relates to user queries that can be answered using a combination of schema and business context 217 (e.g., KPI and data dictionaries). In general, an L2 complexity query can be answered using both database schema information and context of the query. For example, these user queries may include user queries relating to specific field value mentions, aggregate across promotional channels, etc. L3 complexity (e.g., level 3 complexity) relates to user queries that require reasoning and chain-of-thought framework to breakdown the query into multiple sub-queries. For example, these user queries may include user queries relating to KPI calculations, trends over time of derived entities, price aggregation, and correlation calculations. L4 complexity (e.g., level 4 complexity) relates to user queries that are in the strategic and planning realm and require autonomous agents. For example, these user queries may include user queries relating to a combination of queries relating to L1-L3 in a single query.

[0089] A timeframe selector 219 is used to classify the query asked, demands the use of data from the default timeframe or a custom timeframe dependent on the query. To perform this, the query is passed as an input as part of a prompt to the timeframe selector LLM 219 which then determines if the query asked uses the default time frame or the custom timeframe. The custom timeframe may be included in or derived from the query, whereas the default timeframe may be a system setting (which can be provided to the user, either automatically, or on demand). After determining the complexity of the user query, the system may then forward the information based on the complexity assigned. If the user query is L1 , L2, or L3 the information may be sent to the Text-to-SQL LLM 215 for further processing. An optimized prompt is passed through the Text-to-SQL LLM 215 to obtain an SQL query 221 , parsing the query using NLP, with database schema and business context 217 as inputs. Once a valid SQL server query is obtained, it is passed through a database 223 to extract a resultant table 225. If the user query is L4, the information may be sent for separate processing using an L4 agent and further tools.For the erroneous SQL queries, a correction loop is in place for a fixed number of (one or more) iterations where the error message from the current SQL query execution is fed back to the Text-to-SQL LLM 215 in the next iteration, for example along with the question, schema, and context. This can enable a correct answer to be retrieved.

[0090] The resultant table is generated based on the SQL query and / or the pre-defined code snippet. A resultant output, such as a graph, chart, and / or a plot, may then be generated based on the SQL query and / or the pre-defined code snippet. The SQL query and / or the pre-defined code snippet can be executed on a database (e.g., data store 114) of the enterprise system.

[0091] After passing through the Text-to-SQL LLM 215, the last building block is a feature that provides a natural language summary 227 of the SQL query generated in a Text-to-SQL module and / or the pre-defined code snippet and responsible for the final results in the form of tables or charts. The summary 227 may include an explanation of the logic or operations used in the SQL query, as well as the tables and fields. For logic and / or operations, the system 200 may utilize LLM calls, whereas the extraction of table and column names is done with a rule based approach. The summary 227 may be an electronic summary and provided on the user device 112 in an electronic format. The summary 227 can be based on the resultant table and / or the resultant output and may include displaying the SQL query and / or the pre-defined code snippet.

[0092] FIG. 2B depicts an exemplary flowchart using a chained LLM, according to one or more embodiments. Similar to FIG. 2A, as described above, the system 250 may start with a user query 251. The user query 251 is analyzed to be either “TRUE” or “FALSE” based upon a set of predefined criteria. Once determined, the system 250 may send the query 251 to an exception handling model 253 (if TRUE) or to the data selector agent module 255 and ambiguity resolution module 257 (if FALSE), which determines whether the asked question is in context or out of context for the current tool capabilities. The user query 251 is then parsed with information from a vector database 259 and forecast data 261 . The user query may then be determined to be either phase-2 data 263 or phase-4 data 265. If the user query 251 is determined to be phase-2 data 263, like FIG. 2A, the query 251 is passed through a series of LLM modules (e.g., complexity classifier 267, timeframe selector 269, Text-to-SQL 271 , and SQL database 273) before being output back tothe user. If the user query 251 is determined to be phase-4 data 265, like FIG. 2A, the query 251 is passed through a series of LLM modules (e.g., complexity classifier 275, timeframe selector 277, Text-to-SQL 279, and SQL database 281) before being output back to the user. Phase 4 here refers to L4 type questions which uses predictive model and forecasting as opposed to L1-L3 which are answered by text to SQL.

[0093] FIG. 2C depicts an exemplary flowchart 300 using a multi-step chained LLM task flow, according to a modification of one or more embodiments of FIG. 2B. The reader is referred to functionality already described in relation to FIG. 2B, which is not described again here. In FIG. 2C, the user query 301 is fed directly to a dataset selector agent 303, which chooses between MMX Phase-2 305, MMX Phase-4 307 and Forecast 309. Here, MMX Phase-2 305 is selected and OOC / OOD (MMX) 311 , rather than OOC / OOD (Forecast) 313. As in FIG. 2B, the L1 - L3 queries may be handled using text-to-SQL tools. A widget generator may provide the output to the user. The advanced LLM query functionality (forecasting and other reasoning) is shown here as “L4 Agent”, which triggers (an) additional agent(s) for advanced LLM functionality. If such user queries occur, the method pulls relevant solution outcomes (e.g. forecasting results from other models) or LLM reasoning workflow to support such advanced LLM query function. A first forecast module may be followed by a data fetch and then an inference module, such as a SARI MA (Seasonal Autoregressive Integrated Moving Average) inference module. Two SARI MA modules may be provided, one which is also for training and works on specific brands, and the other an overall inference facility. The same widget generator may provide the output to the user.

[0094] FIG. 3 depicts an exemplary flowchart 320 using both Gen Al and nonGen Al components, according to one or more embodiments. Specifically, FIG. 3 depicts a hybrid solution involving both generative Al and non-generative Al components for added versatility in a tool (e.g., approach may be scaled to an enterprise). For example, a system may include a set of persona-based data sources 321 for all individuals within an enterprise. The system may further include one or more data sources 323 that are passed through automated data pre-processing pipeline 325. The system may then take the information from the persona-based data source 321 and the one or more data sources 323 and pass them through a multilevel orchestrator agent 327 (e.g., determining what agent and / or module eachpiece of information is to be placed). A data source agent 329 may determine the original location of each data item and categorize accordingly. A document agent 331 may determine the types of documents for each data item and categorize them accordingly. One or more servers 333 may be employed to route each data item upon request. Once categorized, the system may then send the data items to an SQL code generator 335 for processing. The SQL code generator 335 may include the context aspect as described in FIG. 2A, prior to the user query 201 being passed through the Text-to-SQL LLM 215.

[0095] FIGs. 4A-4D depict an exemplary flowchart and sample user prompts, according to one or more embodiments. FIG. 4A depicts steps of a query answering system 400 at the top of the diagram, and is focused on a context enrichment layer to aid in answering domain specific questions using LLMs. This flowchart may be implemented into the system as described in FIG. 2A. As described in FIG. 2A, the system may pass the user query 401 through a first classifier LLM 403 (e.g., context / domain classifier). The user query 401 may be compared against a database schema 405 prior to the first classifier LLM 403. For example, the user query 401 may be parsed to extract particular information (e.g., synonyms, aliases, etc.) and a determination of particular metrics (e.g., calculation or aggregation rules). The context enrichment layer 407 may be implemented before, during, and / or after the user query 401 is passed through the second classifier LLM 409 (e.g., complexity classifier). The context enrichment layer 407 may include one or more methods of increasing complexity. For example, a first method may include prompt engineering 411 (e.g., low complexity), a second method may include vector embedding 413 (e.g., medium complexity), and a third method may include fine tuning 415 (e.g., high complexity). One or more of the methods may be included in the context enrichment layer 407 to enhance the context and information surrounding the user query. This may include the use of additional database metadata and data dictionaries 417 that may be driven by subject matter experts 419 (SME). With the addition of one or more context enrichment methods, the system may then proceed as described in FIG. 2A entering the third classifier (e.g, Text-to-SQL classifier), so that the output of the context enrichment layer is fed into the Text-to-SQL classifier.

[0096] The third classifier may be used to determine the SQL query and / or a pre-defined code snippet. The pre-defined code snippet can be a small, reusable piece of code that is already written and can be quickly inserted into a program (e.g.,the LLMs for the enterprise system) to perform a specific function. The pre-defined code snippet may, more specifically, include an API call from one of the data model, application output, or agent. The determination of the SQL query and / or the predefined code snippet can be based on the first classifier (e.g., the context of the query) and the second classifier (e.g., the complexity level of the query).

[0097] FIG. 4A is an overview showing how a user question and database schema are fed into a filtering, classification and checking stage which provide entity extraction (dealing with synonyms and aliases and filtering values) and metric derivation (calculation rules and aggregation rules). The next step is context enrichment as described above, leading to an LLM prompt 421 and output 423 followed by output visualization 425. The three context enrichment methods of increasing complexity described above may be used.

[0098] Some exemplary user prompts and responses used in connection with the one or more context enrichment layers within the system as described in FIG. 2A may use additional context in the form of synonyms. For example, FIG. 4B illustrates a user query 426 that is formulated into a prompt 427 for the LLM. The LLM may use a table schema 429 and additional context 431 to generate a response 433 from the prompt 427. FIG. 4C is a code snippet view of filtering 435, aliases 437 and a derived KPI 439. FIG. 4D shows a fist prompt 441 with a first response 443 and a second prompt 445 with a second response 447 with the use of a rule and synonyms.

[0099] FIGs. 5A-5C depict an exemplary flowchart and sample user interface of a knowledge answering engine, according to one or more embodiments. FIG. 5A depicts a problem solving approach 500 using LLMs. The approach 500 may start with a user query 501 and database schema 503. The user query 501 is then automatically broken down into chain-of-thought segments 505. For example, if the use query is “What is our overall profitability, and are there areas where we can improve our margins?” The approach may automatically break-down the query into one or more segments such as “What is our total revenue?”, “What is our total cost?”, “What is our current profit margins?”, “Understand why there are areas of lower profit margins,” and “Explore potential strategies for improving profit margins in these areas.” Generally however, users are not recommended to ask multiple questions in one statement as the readouts may become inconsistent.

[0100] Once broken-down (if necessary), the system may then utilize a context enrichment layer 507. The context enrichment layer 507 may include data dictionaries, SME inputs, and dashboards 509. The context enrichment layer 507 may include clarifying questions 511. For example, depending on the user query 501 , the approach may need additional details to further assess the query 501 and determine one or more solutions. Upon additional input by the user, if desired, the approach may then proceed to pass the information through a series of LLM classifiers 513 as similarly described in FIG. 2A. Alternatively, the LLM classifiers 513 as similarly described in FIG. 2A may be used in parallel with context enrichment or be used in advance of context enrichment. These arrangements may require additional user input or be automatic. The approach may include additional tools 515 prior to outputting a visualization 523 to the user. These tools may include SQL query module 517, additional LLM models 519, and python code modules 521 . Upon outputting a visualization 523 to the user, the user may include additional inputs to further refine or modify the visualization 523.

[0101] FIGs. 5B-5C depict exemplary user interface prompts and visualization in connection with a user query using the approach as described above in FIG. 5A. FIG. 5B shows a response in the form of a graph 525 (of ROI) output in response to a query 527 (on ROI) and a follow-up query 529, about how to optimize the marketing budget. It also shows a summary 531 of the ROI information and some assumptions 533 which will be made for the follow-up query, to optimize the marketing budget. FIG. 5C shows a displayed clarifying question checking if the assumptions for the follow-up query should be modified. At this point, the user has indicated that they want to proceed and the system has produced an optimization which is displayed in graph and tabular form. The summary has been reproduced and enhanced with the optimization strategy proposed. At the bottom of the screen, further queries are suggested.

[0102] FIGs. 6A-6D depict exemplary flowcharts including user interaction with a knowledge answering engine, according to one or more embodiments. FIG. 6A depicts an exemplary flowchart 600 with respect to higher order question and answer processing. All the steps shown in the top line of FIG. 4A are reproduced here. The workflow may start with a user query 601 and database schema 603 that may be passed through a first classifier LLM 605 (e.g., context / domain classifier). The user query 601 may be parsed and / or filtered to determine the context of the user query601 . For example, it may be determined the query relates to a policy violation for an enterprise. Additional information, such as metadata may be checked to determine if additional context is required. For example, if user has access to data requested (e.g., access permissions). The user query 601 may also be classified using few shot learning and / or keywords. After the first classifier 605, the user query 601 may pass through a context enrichment layer 607 as similarly described in FIG. 4A.

[0103] The context enrichment layer 607 may provide additional prompts to avoid hallucinations and pre-determined business rules. The context enrichment layer 607 may be provided in parallel with the series of LLM classifiers 609, 611 as similarly described in FIG. 2A. Upon completion of the one or more LLM classifiers 609, 611 , the workflow may include an output validator 613. The output validator 613 may include periodic manual and subjective evaluation scores. The output validator 613 may include a check for the information contained in the output against what is stored in the cache, in case there is a discrepancy. The output validator 613 may include a check of user persona for outputting information or visualizations 615 in a particular format. FIG. 6B depicts exemplary user interface images of the workflow and user interaction, as well as some of the underlying principles of the system to identify user intent, build bot personality and enable different interaction modes.

[0104] FIG. 6C depicts an exemplary flowchart between the system and a user, using the example query interaction already introduced in Figs. 5B and 5C as an illustrative user flow. The flowchart 630 may start by displaying a prompt 631 to a user 633 to enter a query 635. The query 635 may be entered and passed to a first classifier 637 (e.g., context classifier). Once the context of the query 635 is determined, the flowchart 630 may then proceed to identify the complexity of the query. If the query is too complex, the flowchart may request additional information 639 from the user. The flowchart 630 may then determine if the system is capable 641 of handling the query 635 given the additional context 639. The query 635 may alternatively be classified 643 as L1 645, L2 647, or L3 649 (e.g., described above with reference to FIG. 2A), and the flowchart 630 may proceed to process the query based on the determined complexity. L1 645 may directly proceed to results 651 , L2 647 may include context enrichment 653 and query conversion 655 prior to proceeding to results 651 , and L3 649 may validate assumption with user input 657 and execute code / call external modules 659 prior to proceeding to results 651 .

[0105] Upon completion, the flowchart 630 may then proceed to check if the response answers 661 the user query 635. If “NO”, the flowchart 630 may display a message 663 to the user stating the flowchart 630 has limited capacity and cannot answer. If “YES”, the flowchart 630 may display the response 665 to the user. For example, the displayed response 665 may include visualization highlights 667 that are provided to a LLM output 669 as similarly depicted in FIGs. 5B-5C. The flowchart 630 may include additional prompts or displays 671 to the user 633 for feedback based on the response output. If the user wants to modify the output, additional information may be input by the user for further processing as described above.

[0106] FIG. 6D depicts an exemplary implementation flowchart according to one or more embodiments. The flowchart 700 may include a GenAI engine module 701 , data module 703, and consumption module 705. The GenAI engine module 701 may include one or more LLM models 707 implemented within an Al platform, for example, Azure Al studio. The Azure Al studio may include inference API(s) 709 and foundational and Fine-tuned models 711 for user query analysis. Gen Al engine module 701 may include LLM orchestration module 713, toolkits 715, context enrichment module 717 and Azure functions and App services 719. The LLM orchestration module 713 may include prompts 721 including chain-of-thought agents, question classifiers, and output validators as described above.

[0107] The context enrichment module 717 may be similar to the context enrichment module described in FIG. 4A. The toolkits module 715 may include Text- to-SQL 723, python 725, and external utility API calls 727. The data module 703 may include one or more data sources module 729 and metadata module 731 . The one or more data sources module 729 may include data sets being structured and / or unstructured. The metadata module 731 may include one or more of a dictionary module 733, business rules module 735, enrichment metadata module 737, and fine- tuning datasets module 739. The dictionary and business rules modules may be part of a data catalog 741 . The enrichment metadata and fine-tuning datasets modules may be part of a Text / JSON on Blob storage 743. The consumption module may include a chat interface module 745, dashboard insights module 747, and co-pilot module 749.

[0108] FIGs. 7A-7B depict an exemplary flowchart architecture of a knowledge answering engine, according to one or more embodiments. FIG. 7A depicts an exemplary architecture diagram 800. The architecture 800 may include a datasource 801 . The data source 801 may include information from one or more sources (e.g., enterprise, APIs and files, etc.). The architecture 800 may pass the data source information through an LLM 803 (e.g., Azure OpenAI) and an application database 805 (e.g., PostgreSQL). The LLM 803 may utilize additional services 807 (a library 807 for efficient similarity search, e.g., FAISS) to source multimedia content based on the information provided from the data sources 801. FAISS may send and transmit information to a set of one or more Azure products (e.g., Azure DevOps, Azure Repo, and Azure Pipeline). After the information from the data sources 801 is passed through the LLM 803 and application database 805, the architecture 800 may continue with an application layer 809. The application layer 809 may include an API layer 811 , a frontend 813, container registry 815, and app services 817. The API layer 811 may include tools 819 (e.g., python and FastAPI). The frontend 813 may include React JS 821 (e.g., a library for web and native user interfaces). The architecture 800 may include single sign-on (SSO) integration 823 for each user 825 within a directory 827.

[0109] FIG. 7B depicts an exemplary implementation flowchart according to one or more embodiments. The flowchart 850 may include a GenAI engine module 851 , data module 853, and consumption module 855. The GenAI engine module 851 may include one or more LLM models 857 implemented within an Al platform, e.g., Azure Al studio 859. The Azure Al studio 859 may include inference API(s) 861 and foundational and Fine-tuned models 863 for user query analysis. Gen Al engine module 851 may include LLM orchestration module 863, toolkits 865, context enrichment module 867 and Azure App services 869. The LLM orchestration module 863 may include prompts 871 including chain-of-thought agents, question classifiers, and output validators. The context enrichment module 867 may be similar to the context enrichment module 407 described in FIG. 4A.

[0110] The toolkits module 865 may include Text-to-SQL 873, python 875, and external utility API calls 877. The data module 853 may include a web application metadata module 879, a data sources & data sets module 887, and metadata module 891. The web application metadata module 879 may include user tables 881 , conversations 883, and LLM configurations 885. Data sources & data sets module 887 may include internal and external data sources 889 (e.g., marketing spending, contribution, and return on investments (ROI)). The metadata module 891 may include a dictionary module 893 and a business rules module 895. Thedictionary 893 and business rules 895 modules may be part of Text / JSON on Blob storage 897. The consumption module 855 may include a chat interface module 899, dashboard insights module 901 , and co-pilot module 903. FIG. 7B differs from FIG. 6D primarily in the user of a web application metadata (Postgres) module (shown bottom left) with user table, conversations and LLM configuration sub- modules, and in that the meta data does not include modules of enriched metadata and fine-tuning datasets or the data calatog.

[0111] FIG. 8 depicts an exemplary architecture diagram of a knowledge answering engine, according to one or more embodiments. The architecture 910 may include a data source 911 . The data source 911 may include information from one or more sources (e.g., enterprise, APIs and files, calendars, etc.). The architecture 910 may then pass the information through an SQL server 913. The SQL server 913 may provide the system with the function of storing and retrieving data as requested by other software applications. The architecture 910 may then send the information through indexing 915, vector databases 917, and prompt management 919. Prompt management 919 may include the use of one or more LLMs (e g., Azure MLPrompt Flow, etc.). The use of the one or more LLMs may allow the system to determine context of a user query, select a dataset, determine the complexity of the user query, and translate the user query to an SQL query. Indexing may include the use of third party API’s (e.g., LangChain) that may assist the one or more LLMs. The LLM may utilize additional services (e.g., FAISS) to source multimedia content based on the information provided from the data sources. FAISS may send and transmit information to a set of one or more Azure products (e.g., Azure DevOps, Azure Repo, and Azure Pipeline). The architecture 910 may then send the information through a representation state transfer (REST) API 921 to define a set of constraints for how the architecture 910 of a distributed system should behave. The REST API 921 may emphasize uniform interface, independent deployment of components, scalability of interaction between components, and creating a layered architecture to promote caching to reduce user-perceived latency and enforce security.

[0112] The architecture 910 may then send the information through application services 923 and container registry 925. The application services 923 may include both back-end application services 927 and front-end application services 929. The back-end service 927 may include the use of a programminglanguage (e.g., Python) and supported API (e.g., Fast API). Front-end service 929 may include JavaScript library (e.g., ReactJS). The application services 923 may utilize business logic 931 and guardrails 933. Business logic may include software that has the ability to automate deployment of applications in lightweight containers so that each application may work efficiently in isolation (e.g., Docker). The architecture may include Azure application insights 935. Azure application insights 935 may assist the system in investigating, monitoring, usage, and code analysis. These features may allow the system to perform more efficiently with the ability to handle large amounts of data and metric.

[0113] The architecture 910 may then send the information for consumption. In doing so, the system may utilize reporting and application layers 937. The reporting and application layers 937 may collect and determine the output of information be sent to the user for display as well as kept in storage using version control. The output information may be in the form of text, image, video, audio, graphic, or a combination thereof. The output information may then be adjusted or given feedback from one or more users. The input from the users may modify and update the system based on the input received. For example, the input may be in the form of an additional query to modify the content output. Another example may include the user being a SME (subject matter expert) to update the information output, providing additional content and / or context to the output information for further analysis and storage. The system may include sending and transmitting information between each location and in different orders depending on the information required at any given time.

[0114] FIG. 9 depicts a further exemplary architecture diagram of a knowledge answering engine, according to one or more embodiments. The reader is referred to functionality already described in relation to FIG. 8, which is not described again here. FIG. 9 is separated into source data 951 of various types, and the system environment 953, which includes back-end components 955, pipeline components 957, app service 959 and authentication and security 961. The back-end components 955 include storage 963 (with an Azure cache for Redis, Blob storage, PostgreSQL with a prompt library and chat logging, and an internal data lake), Azure’s Al studio 965, and a vector database 967. The pipeline components 957 include a container registry 969 which is fed via Azure functions 971 and a build pipeline 973 from the BE version control repository 975 and FE version controlrepository 977 in Azure Devops. The container registry 969 feeds the front-end node 979 and back-end node 981 in the app service 959, which also includes app monitoring 983.

[0115] This solution architecture diagram 950 indicates how Source Data 951 components are fed into the MarsGPT LLM application Back-End Components 955, prepared and transformed through storage and database components, leveraging Azure Service’s Al studio for LLM calling and Prompt Flow for prompt management, using Al Search for Vector database indexing preparation, with LLM backend components orchestrated by the Langchain Framework. The entire application then is created with various DevOps sections, including: the app service components 959 for front-end and app monitoring Observability, Pipeline Components 957, and, Authentication and Security 961 . Various work flows are indicated on the bottom of the solution architecture diagram 950. The query DB dataflow is from the SQL DB to the Al studio and Azure cache for Redis as well as between the datalake and backend node and app insights and from the PostgreSQL to the frontend node. The data ingestion flow is from the source data to the SQL DB and to the blob storage, Al search which communicates with the OpenAI suite. The logging flow is from the PostgreSQL and datalake to the front-end node 979 and back-end node 981 respectively and from the front-end node 979 to data insights 983. The DevOps flow is from the BE Repo 975 and FE Repo 977 to the build pipelines 973, via the Azure functions 971 to the container registry 969 and front-end and back-end nodes 979, 981.

[0116] The system performance was tested using a test of input queries of complexity level L1-L4 as set out herein for user acceptance testing. The embodiments disclosed herein give clearly improved functionality over conventional approaches (i.e., prior art), in terms of accuracy and latency and also conciseness and consistency. Table 1 below represents a prior art system.Table 2 below shows the performance characteristics of the embodiments disclosed herein.

[0117] Both tables use the following definitions:

[0118] Accuracy % is the overall absolute correct answer % of a pre-defined evaluation set of questions. Consistency % (also can be called as Reliability / Reproducibility) is out of testing cadences, whether results are consistent. This is a specific evaluation for GenAI LLM, given they are probability based and tend to produce different results over time if parameter settings conditions are set differently. Latency is the time measurement for users receiving responses after completing queries and can be gathered through user testing and / or leveraging anLLM workflow orchestration framework such as LangSmith / langchain. Business questions refers to set of questions received directly from business / end users to test the tool. There were 13 business questions. Conciseness is representation of how concise and interpretable the answer was, like whether tool used right visual or graphs to represent the answer.

[0119] FIG. 10 illustrates an implementation of a computer system that executes techniques presented herein. The computer system 1000 includes a set of instructions that are executed to cause the computer system 1000 to perform any one or more of the methods or computer based functions disclosed herein. The computer system 1000 operates as a standalone device or is connected, e.g., using a network, to other computer systems or peripheral devices.

[0120] Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the specification, discussions utilizing terms such as "processing," "computing," "calculating," “determining”, analyzing” or the like, refer to the action and / or processes of a computer or computing system, or similar electronic computing device, that manipulate and / or transform data represented as physical, such as electronic, quantities into other data similarly represented as physical quantities.

[0121] In a similar manner, the term "processor" refers to any device or portion of a device that processes electronic data, e.g., from registers and / or memory to transform that electronic data into other electronic data that, e.g., is stored in registers and / or memory. A “computer,” a “computing machine,” a "computing platform," a “computing device,” or a “server” includes one or more processors.

[0122] In a networked deployment, the computer system 1000 operates in the capacity of a server or as a client user computer in a server-client user network environment, or as a peer computer system in a peer-to-peer (or distributed) network environment. The computer system 1000 is also implemented as or incorporated into various devices, such as a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile device, a palmtop computer, a laptop computer, a desktop computer, a communications device, a wireless telephone, a land-line telephone, a control system, a camera, a scanner, a facsimile machine, a printer, a pager, a personal trusted device, a web appliance, a network router, switch or bridge, or any other machine capable of executing a set ofinstructions (sequential or otherwise) that specify actions to be taken by that machine. In a particular implementation, the computer system 1000 is implemented using electronic devices that provide voice, video, or data communication. Further, while the computer system 1000 is illustrated as a single system, the term “system” shall also be taken to include any collection of systems or sub-systems that individually or jointly execute a set, or multiple sets, of instructions to perform one or more computer functions.

[0123] As illustrated in FIG. 10, the computer system 1000 includes a processor 1002, e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both. The processor 1002 is a component in a variety of systems. For example, the processor 1002 is part of a standard personal computer or a workstation. The processor 1002 is one or more processors, digital signal processors, application specific integrated circuits, field programmable gate arrays, servers, networks, digital circuits, analog circuits, combinations thereof, or other now known or later developed devices for analyzing and processing data. The processor 1002 implements a software program, such as code generated manually (i.e., programmed).

[0124] The computer system 1000 includes a memory 1004 that communicates via bus 1008. Memory 1004 is a main memory, a static memory, or a dynamic memory. Memory 1004 includes, but is not limited to, computer-readable storage media such as various types of volatile and non-volatile storage media, including but not limited to random access memory, read-only memory, programmable read-only memory, electrically programmable read-only memory, electrically erasable read-only memory, flash memory, magnetic tape or disk, optical media and the like. In one implementation, the memory 1004 includes a cache or random-access memory for the processor 1002. In alternative implementations, the memory 1004 is separate from the processor 1002, such as a cache memory of a processor, the system memory, or other memory. Memory 1004 is an external storage device or database for storing data. Examples include a hard drive, compact disc (“CD”), digital video disc (“DVD”), memory card, memory stick, floppy disc, universal serial bus (“USB”) memory device, or any other device operative to store data. The memory 1004 is operable to store instructions executable by the processor 1002. The functions, acts, or tasks illustrated in the figures or described herein are performed by processor 1002 executing the instructions stored in memory 1004. Thefunctions, acts, or tasks are independent of the particular type of instruction set, storage media, processor, or processing strategy and are performed by software, hardware, integrated circuits, firmware, micro-code, and the like, operating alone or in combination. Likewise, processing strategies include multiprocessing, multitasking, parallel processing, and the like.

[0125] As shown, the computer system 1000 further includes a display 1010, such as a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid-state display, a cathode ray tube (CRT), a projector, a printer or other now known or later developed display device for outputting determined information. The display 1010 acts as an interface for the user to see the functioning of the processor 1002, or specifically as an interface with the software stored in the memory 1004 or in the drive unit 1006.

[0126] Additionally or alternatively, the computer system 1000 includes an input / output device 1012 configured to allow a user to interact with any of the components of the computer system 1000. The input / output device 1012 is a number pad, a keyboard, a cursor control device, such as a mouse, a joystick, touch screen display, remote control, or any other device operative to interact with the computer system 1000.

[0127] The computer system 1000 also includes the drive unit 1006 implemented as a disk or optical drive. The drive unit 1006 includes a computer- readable medium 1022 in which one or more sets of instructions 1024, e.g. software, is embedded. Further, the sets of instructions 1024 embodies one or more of the methods or logic as described herein. Instructions 1024 resides completely or partially within memory 1004 and / or within processor 1002 during execution by the computer system 1000. The memory 1004 and the processor 1002 also include computer-readable media as discussed above.

[0128] In some systems, computer-readable medium 1022 includes the set of instructions 1024 or receives and executes the set of instructions 1024 responsive to a propagated signal so that a device connected to network 1030 communicates voice, video, audio, images, or any other data over network 1030. Further, the sets of instructions 1024 are transmitted or received over the network 1030 via the communication port or interface 1020, and / or using the bus 1008. The communication port or interface 1020 is a part of the processor 1002 or is a separate component. The communication port or interface 1020 is created in software or is aphysical connection in hardware. The communication port or interface 1020 is configured to connect with the network 1030, external media, display 1010, or any other components in the computer system 1000, or combinations thereof. The connection with network 1030 is a physical connection, such as a wired Ethernet connection, or is established wirelessly as discussed below. Likewise, the additional connections with other components of the computer system 1000 are physical connections or are established wirelessly. Network 1030 alternatively be directly connected to the bus 1008.

[0129] While the computer-readable medium 1022 is shown to be a single medium, the term "computer-readable medium" includes a single medium or multiple media, such as a centralized or distributed database, and / or associated caches and servers that store one or more sets of instructions. The term "computer-readable medium" also includes any medium that is capable of storing, encoding, or carrying a set of instructions for execution by a processor or that causes a computer system to perform any one or more of the methods or operations disclosed herein. The computer-readable medium 1022 is non-transitory, and may be tangible.

[0130] The computer-readable medium 1022 includes a solid-state memory such as a memory card or other package that houses one or more non-volatile readonly memories. The computer-readable medium 1022 is a random-access memory or other volatile re-writable memory. Additionally or alternatively, the computer- readable medium 1022 includes a magneto-optical or optical medium, such as a disk or tapes or other storage device to capture carrier wave signals such as a signal communicated over a transmission medium. A digital file attachment to an e-mail or other self-contained information archive or set of archives is considered a distribution medium that is a tangible storage medium. Accordingly, the disclosure is considered to include any one or more of a computer-readable medium or a distribution medium and other equivalents and successor media, in which data or instructions are stored.

[0131] In an alternative implementation, dedicated hardware implementations, such as application specific integrated circuits, programmable logic arrays, and other hardware devices, is constructed to implement one or more of the methods described herein. Applications that include the apparatus and systems of various implementations broadly include a variety of electronic and computer systems. One or more implementations described herein implement functions using two or more specific interconnected hardware modules or devices with related control and datasignals that are communicated between and through the modules, or as portions of an application-specific integrated circuit. Accordingly, the present system encompasses software, firmware, and hardware implementations.

[0132] Computer system 1000 is connected to network 1030. Network 1030 defines one or more networks including wired or wireless networks. The wireless network is a cellular telephone network, an 802.10, 802.16, 802.20, or WiMAX network. Further, such networks include a public network, such as the Internet, a private network, such as an intranet, or combinations thereof, and utilizes a variety of networking protocols now available or later developed including, but not limited to TCP / IP based networking protocols. Network 1030 includes wide area networks (WAN), such as the Internet, local area networks (LAN), campus area networks, metropolitan area networks, a direct connection such as through a Universal Serial Bus (USB) port, or any other networks that allows for data communication. Network 1030 is configured to couple one computing device to another computing device to enable communication of data between the devices. Network 1030 is generally enabled to employ any form of machine-readable media for communicating information from one device to another. Network 1030 includes communication methods by which information travels between computing devices. Network 1030 is divided into sub-networks. The sub-networks allow access to all of the other components connected thereto or the sub-networks restrict access between the components. Network 1030 is regarded as a public or private network connection and includes, for example, a virtual private network or an encryption or other security mechanism employed over the public Internet, or the like.

[0133] In accordance with various implementations of the present disclosure, the methods described herein are implemented by software programs executable by a computer system. Further, in an example, non-limited implementation, implementations can include distributed processing, component / object distributed processing, and parallel processing. Alternatively, virtual computer system processing can be constructed to implement one or more of the methods or functionality as described herein.

[0134] Although the present specification describes components and functions that are implemented in particular implementations with reference to particular standards and protocols, the disclosure is not limited to such standards and protocols. For example, standards for Internet and other packet switched networktransmission (e.g., TCP / IP, LIDP / IP, HTML, HTTP) represent examples of the state of the art. Such standards are periodically superseded by faster or more efficient equivalents having essentially the same functions. Accordingly, replacement standards and protocols having the same or similar functions as those disclosed herein are considered equivalents thereof.

[0135] It will be understood that the steps of methods discussed are performed in one embodiment by an appropriate processor (or processors) of a processing (i.e., computer) system executing instructions (computer-readable code) stored in storage. It will also be understood that the disclosure is not limited to any particular implementation or programming technique and that the disclosure is implemented using any appropriate techniques for implementing the functionality described herein. The disclosure is not limited to any particular programming language or operating system.

[0136] It should be appreciated that in the above description of example embodiments of the present disclosure, various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of the present disclosure, however, is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of the present disclosure.

[0137] Furthermore, while some embodiments described herein include some but not other features included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the present disclosure, and form different embodiments, as would be understood by those skilled in the art. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0138] Furthermore, some of the embodiments are described herein as a method or combination of elements of a method that can be implemented by a processor of a computer system or by other means of carrying out the function. Thus, a processor with the necessary instructions for carrying out such a method orelement of a method forms a means for carrying out the method or element of a method. Furthermore, an element described herein of an apparatus embodiment is an example of a means for carrying out the function performed by the element for the purpose of carrying out the present disclosure.

[0139] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the present disclosure are practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0140] Thus, while there has been described what are believed to be the preferred embodiments of the present disclosure, those skilled in the art will recognize that other and further modifications are made thereto without departing from the spirit of the present disclosure, and it is intended to claim all such changes and modifications as falling within the scope of the present disclosure. For example, any formulas given above are merely representative of procedures that may be used. Functionality may be added or deleted from the block diagrams and operations may be interchanged among functional blocks. Steps may be added or deleted to methods described within the scope of the present disclosure.

[0141] The above disclosed subject matter is to be considered illustrative, and not restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other implementations, which fall within the true spirit and scope of the present disclosure. Thus, to the maximum extent allowed by law, the scope of the present disclosure is to be determined by the broadest permissible interpretation of the following claims and their equivalents, and shall not be restricted or limited by the foregoing detailed description. While various implementations of the disclosure have been described, it will be apparent to those of ordinary skill in the art that many more implementations and implementations are possible within the scope of the disclosure. Accordingly, the disclosure is not to be restricted except in light of the attached claims and their equivalents.

Claims

What is claimed is:1 . A method comprising: displaying, on a user device, a prompt to a user to query an enterprise system; receiving, from the user device, an input from the user, wherein the input is a query to the enterprise system; determining, using a first classifier, a context of the query; determining, using a second classifier, a complexity level of the query; determining, using a third classifier, an SQL query and / or a code snippet based on the context of the query and the complexity level of the query; generating at least one of a resultant table and a resultant output based on the SQL query and / or the code snippet and executing the SQL query and / or the code snippet on a database of the enterprise system; and generating and outputting, on the user device, an electronic summary relating to the SQL query and / or the code snippet of the enterprise system and / or based on the resultant table and / or the resultant output, wherein the electronic summary preferably further includes displaying at least one of the SQL query or the code snippet.

2. The method of claim 1 , wherein the code snippet includes an API call from at least one data model, an application output, or an agent.

3. The method of claim 1 or 2, wherein the first classifier includes a first LLM that classifies the query based on a validity of the context using database schema and business context.

4. The method of claim 3, wherein the first classifier rejects the query as out of scope when it is classified as having an invalid context and does not pass to a next layer.

5. The method of any of the preceding claims, further comprising context enrichment to enhance the context and information surrounding the user query, thecontext enrichment preferably including use of database metadata and database dictionaries and / or clarifying questions posed to the user.

6. The method of any of the preceding claims, wherein determining the context of the query includes use of a persona-based data source and / or a custom access data source for individuals within the enterprise system, as well as one or more other data sources.

7. The method of claim 6, wherein information from the persona-based data source and the one or more other data sources is passed through an orchestrator agent to determine where each piece of information is to be placed, wherein a data source agent determines an original location of each data item and categorizes it accordingly, each data item then being sent to an SQL code generator for processing.

8. The method of any of the preceding claims, wherein determining the complexity level of the query using the second classifier further includes: determining at least one of a level 1 complexity, a level 2 complexity, a level 3 complexity, or a level 4 complexity.

9. The method of claim 8, wherein the level 1 complexity is answered using only database schema information, wherein the level 2 complexity is answered using the database schema information and the context of the query, wherein the level 3 complexity is answered using a chain-of-thought framework to break down the query into sub-queries, and wherein the level 4 complexity is answered using one or more autonomous agents.

10. The method of any of the preceding claims, further comprising requesting additional information from the user if the query has a complexity level above a complexity threshold and / or to avoid hallucinations and / or to satisfy predetermined business-rules.11 . The method of claim 8, 9 or 10, wherein a level 4 complexity query is broken down into queries of lower complexity level and / or handled in a separate toolfrom level 1 , level 2, and level 3 queries, for example by calling a simulator or forecasting tool.

12. The method of any of the preceding claims, further comprising selecting a timeframe for use with the query using a timeframe selector which determines a default timeframe or custom timeframe to be used.

13. The method of any of the preceding claims, wherein the third classifier includes a text-to-SQL LLM.

14. The method of any of the preceding claims, wherein determining, using the third classifier, the SQL query includes using at least one or natural language programming (NLP), natural language processing, or Large Language Model.

15. The method of any of the preceding claims, wherein any SQL query error message is fed back through the third classifier in a next iteration along with the query, database schema, and the context of the query.

16. The method of any of the preceding claims, further comprising displaying the resultant output on the user device and preferably also displaying any assumptions used in arriving at the resultant output or in arriving at a next resultant output.

17. The method of any of the preceding claims, further including displaying an additional prompt to the user after outputting, on the user device, the resultant output and / or the electronic summary.

18. The method of claim 17, further comprising running an improved query in response to a user response to the additional prompt.

19. The method according to any of the preceding claims, wherein the resultant output is subject to evaluation, preferably according to at least one of a check for information contained in the resultant output against what is stored incache and a check of user persona for outputting information or visualizations in a particular format.

20. A system comprising: a memory configured to store instructions; and one or more processors configured to execute the instructions to perform operations according to any of the preceding claims.21 . A computer program, or a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising the steps of any of claims 1- 19.

Images