Hybrid AI architecture for natural language queries

Abstract

An example scheme performs natural language query processing on a mixed utterance. Precise segments are identified within the mixed utterance and processed with a symbolic AI interpreter configured to generate a first interpretation. Within the mixed utterance, precise segments are replaced with placeholder items, resulting in a blurred utterance. The fuzzy utterance is processed with a statistical AI interpreter configured to generate a second interpretation. The first interpretation is merged with the second interpretation using the mixed utterance into a merged template and a placeholder item is used as a location of the first interpretation within the second interpretation. A complete interpretation is generated and sent to a query generator.

Description

Background Technology

[0001] Natural Language Query (NLQ) systems are computer systems that facilitate user interaction with data sources (e.g., databases, repositories, knowledge bases) using queries or questions in natural human language (e.g., English, French, German, etc.). The primary goal of such systems is to enable users to retrieve information, seek answers, or perform specific tasks by expressing their needs in human language (“discourse,” whether written or spoken) rather than requiring them to use a specific query language or structured commands. For example, in the context of a business sales database, a user could enter the query: “What were the top five liquor sales customers in Chicago last week?” An NLQ system could then translate this natural language discourse into a structured computer query and apply it to the target data source.

[0002] Historically, this translation process has employed one of two competing technologies: symbolic artificial intelligence (AI) interpreters or statistical AI interpreters. Symbolic AI interpreters attempt to solve the translation problem using computational language techniques such as word segmentation, lexicalization, syntactic parsing, or semantic interpretation, as well as algorithmic structured query generation. Statistical AI interpreters attempt to solve the translation problem using statistical AI techniques such as deep neural networks or large language models (LLMs).

[0003] Symbolic AI and statistical AI approaches in NLQ each have their own advantages and disadvantages. Symbolic AI typically provides highly predictable and consistent results, where the specific reasoning behind the choices made in the interpretation is interpretable and easily configurable by the user of the system. Furthermore, symbolic AI generally performs particularly well in translating precise wording problems of varying complexity into correct structured computer queries. However, they tend to fall short in handling poor grammar, unwanted terminology, and vague or imprecise word-based discourse, which requires significant reasoning or complex clarification.

[0004] On the other hand, some statistical AI methods (particularly LLM) perform quite well in handling complex clarification and reasoning guided by humans, as well as adapting to new vocabulary and unusual syntactic structures. However, their inherent flexibility has some drawbacks. That is, statistical AI methods, by their very nature, are less predictable, less consistent, difficult to customize, and lack the ability to explain how their answers were derived. Furthermore, as seen in the “illusion” of LLM in recent years, the reliability and correctness of such systems can vary widely from discourse to discourse. Summary of the Invention

[0005] The disclosed examples are described in detail below with reference to the accompanying figures. The following overview is provided to illustrate some of the examples disclosed herein. However, this is not to imply that all examples are limited to any particular configuration or sequence of operations.

[0006] Example schemes for performing natural language query processing on mixed discourse include: receiving mixed discourse, which is a natural language query request for data from a data source; identifying a first precise segment within the mixed discourse; processing the first precise segment using a symbolic artificial intelligence (AI) interpreter configured to generate a first interpretation; replacing the first precise segment within the mixed discourse with placeholder items to generate a first ambiguous discourse; processing the first ambiguous discourse using a statistical AI interpreter configured to generate a second interpretation; merging the first and second interpretations, using the mixed discourse as a template for merging and placeholder items as positions for the first interpretation within the second interpretation to generate a complete interpretation; and sending the complete interpretation to a query generator configured to generate at least one database query against the data source to generate query results displayed in response to the natural language query request. Attached Figure Description

[0007] The disclosed example is described in detail below with reference to the accompanying drawings:

[0008] Figure 1 An example architecture of a Natural Language Query (NLQ) system is shown, which advantageously provides natural language query processing for precise, fuzzy, and mixed discourse;

[0009] Figure 2 It shows things like Figure 1 A diagram of component-level details and data flow within an example mixed interpreter;

[0010] Figure 3 It shows things like Figure 1 A diagram illustrating component-level details and data flow within an example statistical artificial intelligence (AI) interpreter;

[0011] Figure 4 This is a flowchart illustrating exemplary operations that can be performed by an NLQ system to provide a hybrid architecture in natural language query processing; and

[0012] Figure 5 It is a block diagram of an example computing device (e.g., a computer storage device) for implementing the aspects disclosed herein, and is generally designated as a computing device.

[0013] Throughout the accompanying drawings, corresponding reference numerals denote corresponding parts. Any drawings may be combined to form a single example or embodiment. Detailed Implementation

[0014] Some Natural Language Query (NLQ) systems have attempted to leverage both statistical and symbolic methods, either through simple union or by using statistical artificial intelligence (AI) as an application programming interface (API). In simple union, the symbolic AI NLQ system and the statistical AI NLQ system execute in parallel, with a pre-classifier selecting which NLQ system to use for any given utterance, or a post-processing scoring mechanism selecting which competing outcome is more likely to be correct. However, the simple union approach has at least two drawbacks. First, both the pre-classifier and the post-processing scoring technique are fragile and unreliable because they require predicting / evaluating the behavior of two fundamentally different systems. Second, union cannot account for mixed problems (e.g., utterances with some precise components and some ambiguous components).

[0015] In some recent approaches, statistical AI manages the translation of discourse and has the ability to invoke symbolic AI as a subsystem for discourse parts that may be processed symbolically. While this approach interprets some mixed discourse, it comes at the cost of predictability, as accurate understanding through the application of symbolic AI depends on the unpredictability and variability of statistical AI systems.

[0016] These existing solutions produce poor results, forcing users to reformulate and resubmit their queries, thus wasting computing power, storage, and network bandwidth.

[0017] Conversely, in some examples, the NLQ system described in this paper provides a hybrid approach that leverages the strengths of both statistical and symbolic AI methods, using each to mitigate the weaknesses of the other. The NLQ system includes a unifier module that uses a symbolic AI interpreter to handle exact queries, a statistical AI interpreter to handle fuzzy queries, and a hybrid interpreter that handles queries that include some combination of exact and fuzzy query components.

[0018] Upon receiving a new utterance (e.g., a query formatted in natural language provided by a user), the conjoiner module evaluates the utterance to determine how to process it. For utterances that are purely exact queries (e.g., where all sub-components of the utterance can be precisely mapped to specific components of (multiple) target data sources), the conjoiner module directs those utterances to the symbolic AI interpreter for processing. For utterances that are purely fuzzy queries (e.g., without sub-components that can be precisely mapped to specific components of (multiple) target data sources), the conjoiner module directs those utterances to the statistical AI interpreter for processing. For utterances that include some exact sub-components and some fuzzy sub-components (“mixed utterances”), the conjoiner module directs those mixed utterances to the mixed interpreter for processing.

[0019] The hybrid interpreter resolves a mixed discourse into precise sub-components (“(multiple) partially precise discourse”) and fuzzy sub-components (“(multiple) partially fuzzy discourse”). The partially precise discourse is processed by a symbolic AI interpreter to generate (multiple) precise interpretations (“(multiple) partially precise interpretations”) (e.g., the precise mapping of a database query to a specific data component). To address the fuzzy sub-components of the discourse, the hybrid interpreter includes a discourse rewriter module that extracts the precise components from the complete discourse (e.g., because they will be addressed by the symbolic AI interpreter) and rewrites the discourse into “reduced discourse” (e.g., purely fuzzy discourse). This reduced discourse is processed by a statistical AI interpreter to generate “reduced discourse interpretations.” The (multiple) partially precise interpretations and (multiple) reduced discourse interpretations are sent along with the original discourse to an interpretation merging module. The interpretation merging module combines the (multiple) partially precise interpretations with the (multiple) reduced discourse interpretations, using the original discourse as a template for reorganization. This merging results in one or more “complete interpretations,” each of which represents a more refined form of the discourse tailored to the target data source based on the original natural language discourse.

[0020] These complete explanations can then be sent to a ranking module, which scores each explanation based on its likelihood of correctly representing the discourse intent. The top-ranked explanations (multiple) are sent to a query generator module, which converts the selected explanations into a database query specific to the target data source. This database query is sent to the target data source to generate an output set, which can then be sent to the user for display.

[0021] In some examples, the NLQ system also provides a utterance rewriter module that uses LLM combined with linguistic metadata from the target data source to rewrite purely ambiguous utterances into precise utterances. The utterance rewriter module allows user participation in the dialogue (e.g., suggesting clarification, requesting user input). To make target utterances that a symbolic AI interpreter might be able to process a target, the LLM is prepared using linguistic information about the target data source and built-in system capabilities as context (e.g., database schema, data instance terminology information matching the utterance, vocabulary corresponding to the symbolic AI interpreter's core system capabilities, etc.). Therefore, the large language model generates one or more "candidate precise utterances," which can be sent to the symbolic AI interpreter for processing. Where linguistic metadata can be used to rewrite ambiguous utterances into precise utterances, these precise utterances are then processed by the symbolic AI interpreter to generate an interpretation (e.g., instead of a statistical AI interpreter). The linguistic metadata associated with the target data source can also be used by the symbolic AI interpreter to interpret each precise utterance in the precise utterances generated by the LLM.

[0022] The example scheme uses NLQ processing techniques to improve the processing performance and results of mixed utterances. These techniques leverage the respective strengths of statistical AI interpreters and symbolic AI interpreters in interpreting purely ambiguous and purely precise utterances, respectively. The example scheme provides a technical advantage over existing methods by, for example, identifying precise segments(s) within the mixed utterance and processing those precise segments using a symbolic AI interpreter. Furthermore, precise segments are replaced within the mixed utterance to generate purely ambiguous utterance, which is then processed using a statistical AI interpreter. Each of the precise and ambiguous interpretations is then merged to produce a complete interpretation. The example scheme improves computational processing by, for example, providing results that are more likely to be the query originally requested by the user, as the precise portion of their query is processed by an interpreter more suited to precise queries, and the ambiguous portion of their query is processed by another interpreter more suited to ambiguous queries.

[0023] Various examples are described in detail with reference to the accompanying drawings. In any preferred embodiment, the same reference numerals are used in all drawings to refer to the same or similar parts. References to specific examples and implementations throughout this disclosure are provided for illustrative purposes only and are not intended to limit all examples unless indicated otherwise.

[0024] Figure 1 An example architecture of an NLQ system 100 is shown, which advantageously provides natural language query processing for precise, fuzzy, and mixed utterances. In the NLQ system 100, a user 102 wishes to collect some data from a data source or model, such as a target database 160. The target database 160 may be, for example, a corporate database storing various types of information about the operations of a business (e.g., sales data, research data, order data, etc.). However, the user 102 may not know how to formulate a query against the database 160 (e.g., in the query language of the database 160) that will result in the desired outcome. Thus, an NLQ computing device (or simply an "NLQ device") 120 is provided, which can translate natural language utterances (or simply "utterances") 110 into queries against the target database 160 (e.g., as a database query 152).

[0025] This paper provides several examples in natural human language (e.g., English). These examples use natural language to represent the raw, natural language-based input (e.g., utterance 110) provided by user 102 to NLQ system 100, since this raw input is provided in natural language (e.g., written or spoken utterance 110). Furthermore, and for the purpose of comprehensibility, the examples provided herein also use natural language to represent some intermediate processing examples. It should be understood that NLQ device 120 can implement various natural language processing (NLP) techniques to represent natural language utterance 110 and is sufficient to perform the processing steps described herein, such as formal grammars (e.g., context-free grammars, dependency grammars, etc.), parse trees (e.g., selection parse trees, dependency trees, etc.), part-of-speech (POS) tagging, transformation-based parsing, feature structures, syntactic patterns and templates, first-order predicate logic, embeddings, etc. While these intermediate processing steps may be internally represented by some NLP structure or representation of the semantics of natural language terms, phrases, or sentences within NLQ device 120, the intermediate examples are presented in natural language in this paper to aid understanding.

[0026] In this example, NLQ device 120 provides a unifier module 130 and an interpreter set, which are configured to receive and interpret data questions from user 102. The data question is in... Figure 1 The term 110 is used here to represent a written or spoken query in natural language. A data question is any 110 in which the user's intent is to find an answer to a specific question that they believe can be answered from the data available to the NLQ system. The terms "data question," "question," and "discourse" are used interchangeably herein.

[0027] These questions can range from super-specific, single-outcome questions, such as “How many Chicago customers bought more than 3 pounds of rice from Store 6 last month?”, to requests for specific charts, such as “I want to show a line graph of Store 6’s average annual rice sales trend,” and questions requiring complex analysis, such as “What is the best predictor of rice sales?”

[0028] A key characteristic of data questions is their precision. At one end of the spectrum are precise questions, where each part of the question corresponds to specific data existing in (or that can be derived from) the target database 160. For example, "How many customers are there in each state?" can be directly mapped to row counts in a customer table, grouped by the state column, and displayed in an appropriate visualization such as a map, bar chart, or list.

[0029] At the other end of the spectrum are fuzzy problems. Fuzzy problems are those that require analysis or interpretation, which rely on facts or assumptions that may not exist in the problem or data. For example, "What is happening in sales?" does not specify what aspect of sales user 102 wants to see. User 102 could be interested in any number of things, such as factors that help increase sales, or analysis of outliers in sales by region, or simply a chart showing sales over time.

[0030] Whether a question is precise, vague, or somewhere in between may not be an inherent property of the question itself. Rather, this "precision" of the question is a factor in the relationship between the question and the content of the target database. For example, consider the question: "What is the customer lifetime value for our most loyal customers?" In a typical sales database, this question is vague, requiring inferences about how to interpret the lifetime value of each customer (e.g., total sales, average net profit, minimum number of complaints per order) and how to quantify their loyalty (e.g., average orders per year, longest duration since the first order, minimum number of orders placed with competitors). On the other hand, in a database where the customer table includes both a "lifetime value" column and a "loyalty" column, this question would be precise.

[0031] Furthermore, it should be noted that vague questions can differ from ambiguous questions. For example, the question "count by city" might be vague due to the ambiguity of the word "city." The term could refer to the customer's city, the store's city, or the delivery city. However, due to the ambiguity of the term, it is still a precise question with three possible accurate interpretations.

[0032] Problems that fall on the spectrum between purely exact problems and purely fuzzy problems are hybrid problems. Hybrid problems are decomposed by the NLQ device 120 into a combination of exact and fuzzy components. For example, consider the initial problem “What has been happening in the sales of discontinued products in the Eastern region over the past 3 months?” This problem might be considered as a mixture of “what is happening in the sales process” (which is the fuzzy component) and “the sales of discontinued products in the Eastern region over the past 3 months” (which is the exact component).

[0033] How the NLQ device 120 handles a particular data problem (e.g., discourse 110) depends on where discourse 110 falls on the precise / fuzzy spectrum.

[0034] In the example, the concatenation module 130 sends utterance 110 to each of the symbolic AI interpreter 132, the statistical AI interpreter 136, and the hybrid interpreter 134 for processing. Thus, given utterance 110 as input, each of these interpreters 132 through 136 generates interpretations 138A through 138C, respectively. In other examples, the concatenation module 130 analyzes utterance 110 and selects which interpreter 132 through 136 to send utterance 110. More specifically, the concatenation module 130 determines whether utterance 110 is a purely precise problem, a purely fuzzy problem, or a hybrid problem (e.g., a composite problem including at least one fuzzy component and at least one precise component). If utterance 110 is purely precise, the concatenation module 130 passes utterance 110 to the symbolic AI interpreter 132. If utterance 110 is purely fuzzy, the concatenation module 130 passes utterance 110 to the statistical AI interpreter 136. If utterance 110 is a mixed problem, then the combiner module 130 passes utterance 110 to the mixed interpreter 134.

[0035] Symbolic AI interpreter 132 is an interpreter that uses explicit representations of knowledge and symbolic reasoning processes (e.g., manipulating symbols, logical reasoning, predetermined rules) to understand questions posed in natural language. For example, symbolic AI interpreter 132 may be a model that uses one or more formal grammars, semantic frameworks, ontology, rule-based systems, predicate logic, lambda calculus, Prolog and logic programming, concept dependency theory, case-based reasoning, scripts, and patterns or topic roles to generate output (e.g., interpreting 138A). Symbolic AI interpreter 132 is most efficient at handling precise utterances (e.g., compared to handling mixed or ambiguous utterances).

[0036] The statistical AI interpreter 136 is an interpreter that uses probabilistic models and data-driven techniques to interpret and understand questions posed in natural language. For example, the statistical AI interpreter 136 may use one or more of the following: statistical machine translation (SMT) models, statistical language models (LM) (e.g., large language models (LLM)), recurrent neural networks (RNN), long short-term memory (LSTM) networks, attention mechanisms and transformer models (e.g., BERT, GPT), sequence-to-sequence models, embedding models, pre-trained language models (e.g., GPT-3), etc.

[0037] Hybrid interpreter 134 uses both symbolic AI interpreter 132 and statistical AI interpreter 136 to interpret and process utterance 110. More specifically, hybrid interpreter 134 resolves utterance 110 into precise and ambiguous components, and then uses symbolic AI interpreter 132 to generate intermediate interpretations for the precise components and statistical AI interpreter 136 to generate intermediate interpretations for the ambiguous components. These intermediate interpretations are then combined (e.g., merged) to generate interpretation 138B. Additional details about hybrid interpreter 134 are detailed below regarding... Figure 2 Described.

[0038] Interpretations 138A to 138C represent transformations or alterations that the NLQ system 100 can understand and apply to utterance 110. In other words, interpretations 138A to 138C concern possible variations in how the system 100 can interpret utterance 110 provided by user 102 (e.g., in the context of target database 160). In some examples, interpretations 138A to 138C may be provided within an internal structured representation (e.g., as a semantic representation, a non-natural language-based representation that the NLQ system 100 can understand).

[0039] In this example, the input to each of interpreters 132, 134, and 136 is utterance 110, and when given a particular utterance 110, prompting interpreters 132 through 136 generate interpretations 138A, 138B, and 138C, respectively. Although utterance 110 is natural language input (or some syntactic representation thereof), interpretations 138A through 138C generated by interpreters 132 through 136 represent one or more variations on how to interpret utterance 110. Each interpreter 132 through 136 can generate zero or more interpretations 138A through 138C based on a given utterance 110. Each of the generated interpretations 138A through 138C includes a detailed semantic representation of the meaning of utterance 110 and metadata about a particular interpretation 138A through 138C. The explanation metadata includes the confidence level or confidence score of the specific explanation 138A to the specific explanation 138C, and may also include data such as which words in the input utterance were recognized or understood by the interpreter 132, which words were not recognized, and whether the input utterance is syntactically or semantically good (e.g., from the statistical AI interpreter 132).

[0040] In this example, each of the explanations 138A through 138C is sent to the ranking module 140. The ranking module 140 analyzes the confidence level of each of the generated explanations 138A through 138C to identify one or more selected explanations 142 (e.g., the highest confidence explanation among explanations 138A through 138C generated by interpreters 132 through 136). The utterance 110, which may appear to be a precise data question, can be interpreted relative to the target database 160. However, the resulting explanations can vary significantly in terms of confidence that the system understands correctly. If explanation 138A has high confidence, an answer based on the precise explanation is provided to the user. Note that the NLQ device may request clarification in ambiguous situations (e.g., "orders by name"). If explanation 138A has moderate confidence, an answer based on the precise explanation 138A is considered a candidate possible answer. Additionally, utterance 110 is interpreted as if it were an ambiguous or mixed data question (e.g., via explanations 138B and 138C). You can provide the user with any (or both) of the candidate answers based on relative confidence. If the exact interpretation 138A is low confidence, the utterance is considered a fuzzy or mixed data problem. An utterance 110 with an unusually low score can be immediately considered an out-of-domain problem (e.g., no answer is provided).

[0041] After identifying the selected interpretation 142, the NLQ device 120 uses the query generator module 150 to generate one or more database queries 152 based on the selected interpretation 142. These database queries 152 are sent to the target database 160 to generate query results 162, which are then displayed to the user 102 (e.g., via the user's computing device 104). Thus, the NLQ device 120 has obtained a utterance 110 in natural language from the user 102 and generated query results 162 from the target database 160 based on that utterance 110.

[0042] Although Figure 1 The example architecture of the NLQ system 100 shown provides many components within the NLQ device 120 (e.g., a single device providing the various functions described herein), but it should be understood that other architectures are also possible. For example, any of interpreters 132 to 136 or all of them may be provided by other devices or systems (e.g., as standalone models, etc., executed on user computing device 104 via third-party LLMs and APIs, etc.).

[0043] Figure 2 It is shown Figure 1A graph of component-level details and data flow within the example hybrid interpreter 134. While statistical AI interpreters can be used to directly convert fuzzy utterances into target queries (with varying degrees of success), this approach may not achieve the goals of predictability, interpretability, and controllability. To achieve these goals, utterances 110 that are purely fuzzy or otherwise include fuzzy components are converted by the hybrid interpreter 134 into one or more candidate precise utterances for evaluation by the NLQ system 100.

[0044] For example, the vague question "What is the current state of sales?" can be transformed into one or more precise candidate questions such as "What is the status of sales over time?", "Show outliers in sales by region", and "What factors have the greatest impact on total sales?". The process of building candidates may involve asking the user 102 for clarification, such as "What determines the importance of a customer?" The hybrid interpreter 134 may not be able to answer vague questions that require knowledge or data outside the domain.

[0045] The hybrid interpreter 134 processes the hybrid discourse by processing the vague and precise parts separately, and combining their respective results. In the case of a vague discourse with a high-confidence precise part, the precise part can be interpreted, and the vague discourse can be rewritten in a simplified form to be interpreted as a purely vague discourse in the middle.

[0046] For example, in the data question "Why don't green products weighing over 10 kg sell well?", the part about "green products weighing over 10 kg" might have a highly confident, precise explanation. The remaining question, "Why don't some products sell well?", might then be processed into a purely vague statement, perhaps transformed into the precise question, "What factors most affect the total sales of a product?" The queries generated for these two questions can then be combined into the final result.

[0047] Precise data questions with ambiguous parts can also be broken down and reorganized. For example, in the question “Which employees sold products that benefited the company to customers in Chicago,” the span “products that benefited the company” can be processed into an ambiguous utterance, and the result can be combined with a query generated from the simplified precise utterance “Which employees sold products to customers in Chicago.”

[0048] To achieve the functional goals outlined herein, the hybrid interpreter 134 combines the capabilities of both the symbolic AI interpreter 132 (e.g., for precise components) and the statistical AI interpreter 136 (e.g., for fuzzy components). The precise understanding provided by the symbolic AI interpreter 132 offers a deterministic and customizable understanding of precise questions, while the statistical AI interpreter 136 (e.g., via the use of LLM, etc.) is utilized to guide user 102 from fuzzy to precise intents and to expand language coverage when gaps exist in system capabilities or language metadata. The architecture of this NLQ system 100 both encourages user 102 to express precise intents and teaches them how to do so.

[0049] More specifically, in the example, the mixed interpreter 134 (e.g., from...) Figure 1 The concatenation module receives utterance 110. Suppose the original utterance 110 received from user computing device 104 of user 102 is the question: "Which discontinued products under $30 should we restock at the Chicago store?" The utterance 110 includes one or more precise components and one or more vague components.

[0050] The hybrid interpreter 134 includes a discourse parser module 210, which initially examines the original discourse 110 and identifies at least the precise components or fragments of the discourse 110. The precise components of the original discourse 110 are... Figure 2 The segments are identified as partially accurate utterances 214. These partially accurate utterances 214 represent segments of the original utterance 110, which can generate high confidence scores for accurate interpretations. In some examples, the utterance parser module 210 uses a symbolic AI interpreter (e.g., similar to symbolic AI interpreter 132) in conjunction with linguistic metadata about the target database 160 (such as... Figure 3 The language metadata 322 shown is used to interpret discourse 110 to identify which segments of discourse 110 are understood (e.g., can be identified as partially accurate discourse 214 based on language metadata about target database 160) and which segments are not understood (e.g., the remaining segments of discourse 110).

[0051] For example utterance 110, the segment of utterance 110 that is identified by utterance parser module 210 as part of exact utterance 214 is segment " "Discontinued products below 30" and "Chicago store" are used because each of these segments can be precisely mapped to the ingredients in the target database of 160. More specifically, and for example, " The "Discontinued Products (below 30)" section represents a subset of products from the target database 160 (e.g., from a product table that identifies individual products and includes a discontinued status field and a price field). The "Chicago Stores" section represents a subset of stores from the target database 160 (e.g., from a store table that identifies each store by its zip code, along with a zip code table that identifies each zip code by its state, city, municipality, etc.).

[0052] In some examples, the hybrid interpreter 134 requests user 102 to supplement user input 202 (e.g., to clear ambiguity in partially precise utterance 214). For example, the phrase "Chicago stores" might be ambiguous in its scope. One interpretation could be "all stores within the city boundaries of the City of Chicago, Illinois," while another could be "all stores within the 'Chicago land' area" (e.g., Chicago and the surrounding suburbs, or some predetermined area that includes Chicago and the surrounding area). Thus, the hybrid interpreter 134 can identify such ambiguity and prompt user 102 to choose one of the possible interpretations or otherwise refine the partially precise utterance to clear the perceived ambiguity (e.g., by using more explicit language).

[0053] Given the identifiers of the precise utterance 214 and the original utterance 110, the hybrid interpreter 134 also includes a utterance rewriter module 230, which is configured to extract the precise segments from the original utterance 110 and rewrite the original utterance 110 into reduced ambiguous utterance 232. The reduced ambiguous utterance 232 is rewritten by replacing the precise components with placeholders and leaving only the resulting ambiguous segments.

[0054] For example, you can replace "product" with the placeholder. The phrase “discontinued products below 30” (e.g., because this precise phrase represents a subset of products) can be replaced with the placeholder “store” (e.g., because this precise phrase represents a subset of stores). Thus, the original statement 110 can be rewritten as the reduced, vague statement 232 “Which {products} should we restock in {store}?” (wherein curly braces are used here to visually identify the replaced phrase for illustrative purposes).

[0055] Thus, the hybrid interpreter 134 has identified two partially precise utterances 214 and one reduced ambiguous utterance 232. The hybrid interpreter 134 uses the symbolic AI interpreter 132 to generate partially precise interpretations 220 for each partially precise utterance 214 (e.g., similar to how the coordinator module 130 uses the symbolic AI interpreter 132 from...). Figure 1The pure, precise utterance 110 generates an interpretation 138A. Furthermore, the hybrid interpreter 134 uses the statistical AI interpreter 136 to generate a reduced utterance interpretation 240 from the reduced, ambiguous utterance 232 (e.g., similar to how the coordinator module 130 uses the statistical AI interpreter 136 from...). Figure 1 Purely ambiguous discourse 110 generates interpretation 138C).

[0056] Then, the interpretation merging module 250 uses the reduced utterance interpretation 240 and the partially precise interpretation 220 to generate a complete interpretation 252. More specifically, the interpretation merging module 250 uses the original utterance 110 as a template and constructs the complete interpretation 252 by replacing the partially precise interpretations 220 with their corresponding positions within the reduced utterance interpretations 240 (e.g., based on placeholders inserted when creating the reduced ambiguous utterance 232). In some examples, each individual partially precise utterance 214 can produce multiple partially precise interpretations 220. Similarly, each individual reduced ambiguous utterance 232 can result in multiple reduced utterance interpretations 240. Interpreters 132 and 136 may include an internal score for each of the interpretations 220 and 240, and may include a ranking module (e.g., similar to...). Figure 1 The ranking module 140 can filter out low-scoring explanations 220 and 240 (e.g., based on a threshold).

[0057] For example, suppose the vague statement 232, "Which {products} should we restock in {store}?" is transformed into a reduced statement interpretation 240, "Which {product} has the highest price minus cost in each {store}?" Combining this reduced statement interpretation 240 with the partially precise interpretation 220, we get the complete interpretation 252, which is "In each Chicago store, the price is lower than..." Which of the 30 discontinued products have the highest price minus cost?

[0058] Unlike other methods that place the statistical AI interpreter 136 in control of identifying precise parts, the example hybrid interpreter 134 uses the symbolic AI interpreter 132 to pre-interpret precise segments (e.g., because the interpreter 132 is good at handling precise utterances) and only uses the statistical AI interpreter 136 for ambiguous parts, thereby reducing errors that might occur on utterance 110 by using only the symbolic AI interpreter 132 or only the statistical AI interpreter 136.

[0059] Figure 3 It is shown Figure 1The example statistical AI interpreter 136 provides a graph of component-level details and data flow. In this example, the effectiveness of the statistical AI interpreter 136 is improved by using an LLM discourse rewriter module 320, which combines language metadata 322 about the target database 160 with a large language model (LLM) 324 to generate precise questions (e.g., candidate precise discourses 326) from fuzzy discourses.

[0060] The traditional approach to statistical AI interpreters is to train the system by pairing it with a large number of utterances that are expected to be interpreted (or queries). This training enables new utterances to be directly translated into candidate interpretations. However, this approach does not necessarily provide predictability, interpretability, and control.

[0061] In the examples, coupling the features of the LLM 324 with the symbolic AI interpreter 132 offers the opportunity to achieve the best of both worlds, utilizing the inherent flexibility of the LLM 324 to rewrite ambiguity problems (e.g., ambiguous utterance 310) into precise utterances that can be processed by the symbolic AI interpreter 132. In some examples, the original utterance 110 is a purely ambiguous utterance passed to the statistical AI interpreter 136, such as... Figure 1 As shown, and considered as the vague utterance 310 presented here. In other examples, Figure 2 The reduced ambiguity 232 shown is transmitted to, as... Figure 2 The statistical AI interpreter 136 shown is considered as the ambiguous discourse 310 processed by the LLM discourse rewriter module 320.

[0062] In the example, the LLM discourse rewriter module 320 uses language metadata 322 as a primer (“primer”) for the LLM 324 (e.g., mapping terms to objects and relations within the target database 160). The language metadata 322 is configured and edited by the administrator 302 (e.g., via the administrator computing device 304) and can be customized by the administrator 302 based on, for example, a database context such as the structure of the target database 160 (e.g., database schema), the contents of the database 160, information on data instance items matching the discourse, other language pattern information, and vocabulary corresponding to the core system capabilities of the symbol AI interpreter (e.g., “average,” “sorting,” “having,” etc.), some or all of which can be changed based on feedback from the user 102.

[0063] This language metadata 322 can include, for example, simple cases such as mapping the word "customer" to the "CustInfo" table, or mapping to domain-specific terms such as mapping the term "heat transfer coefficient" to the "ingredient.InfHTC" field. Beyond simple term / synonym mappings, language metadata 322 can also define linguistic relationships between objects in database 160. For example, the relationship "an employee sells products to a customer on the order date" is reflected in the "orders" table of database 160, and the relationship between the "products" table and the "price" field can be described as "price indicates how expensive / cheap the product is." By defining synonyms and relationships, model authors have extensive control over how the problem is interpreted within the context of the model.

[0064] In some examples, language metadata 322 can be edited or supplemented based on feedback from user 102. For instance, if user 102 enters the question “Which customers bought cheese last week?” and the statistical AI interpreter 136 generates a response showing which customers returned cheese last week, then LLM 324 does not correctly interpret the word “purchase” in that context. Therefore, administrator 302 can supplement language metadata 322 with the phrase “customers purchase products,” and these transactions are stored in the “orders” table, thus biasing LLM 324 towards using this specific relationship between customers and products, and the source of such data within the target database 160. In this way, a feedback loop is created between the generated output and the improvement to system 100.

[0065] The LLM discourse rewriter module 320, in conjunction with language metadata 322, uses fuzzy discourse 310 as input to prompt the LLM 324 to generate candidate precise discourses(326). These candidate precise discourses(326) can be presented to the submitting user 102 for clarification of fuzzy discourses(326) or for selection of specific candidate precise discourses(326). These one or more candidate precise discourses(326)(326) are then sent to the symbolic AI interpreter 132 to generate candidate interpretations(330). In some examples, the language metadata 322 can also be used by the symbolic AI interpreter 132 (e.g., to interpret each candidate precise discourse(326) generated by the LLM 324), and can be scored, with the best discourse(326) selected as the candidate interpretation(330). These candidate interpretations(330) can then be transformed into interpretations(138C) (e.g., as a response to...). Figure 1 (provided as a response to the purely vague discourse 110) or as a reduced discourse interpretation 242 (e.g., as a response to) Figure 2 (Provided in response to the (multiple) reduced vague statements 232).

[0066] Figure 4This is a flowchart 400 illustrating exemplary operations that can be performed by the NLQ system 100 to provide a hybrid architecture in natural language query processing. In some examples, the operations of flowchart 400 can be similar to... Figure 2 and Figure 3 The operations described in flowchart 400. In the example implementation, the operations of flowchart 400 are performed by... Figure 1 The NLQ device 120 is executed. At operation 410, the NLQ device 120 receives a mixed utterance (e.g., utterance 110), which is a natural language query request for data from a data source (e.g., target database 160). At operation 412, the NLQ device 120 identifies a first precise segment (e.g., partially precise utterance 214) within the mixed utterance. At operation 414, the NLQ device 120 utilizes a symbolic artificial intelligence (AI) interpreter (e.g., symbolic AI interpreter 132) to process the first precise segment, which is configured to generate a first interpretation (e.g., partially precise interpretation 220).

[0067] In this example, at operation 416, NLQ device 120 replaces the first precise segment within the mixed utterance with placeholder entries to generate a first ambiguous utterance (e.g., reduced ambiguous utterance 232). At operation 418, NLQ device 120 processes the first ambiguous utterance using a statistical AI interpreter (e.g., statistical AI interpreter 136), which is configured to generate a second interpretation (e.g., reduced utterance interpretation 240). In some examples, processing the first ambiguous utterance using a statistical AI interpreter further includes: submitting a cue word to a large language model (LLM) (e.g., LLM 324) that includes the first ambiguous utterance and at least one component of language metadata associated with a data source; receiving one or more candidate precise utterances from the LLM; processing the candidate precise utterances using a symbolic AI interpreter to generate one or more candidate interpretations; and selecting one of the candidate interpretations as the second interpretation. In some examples, processing the first fuzzy utterance using a statistical AI interpreter further includes: displaying candidate precise utterances on a user's computing device, and receiving user input indicating a selected candidate precise utterance, wherein processing the candidate precise utterance includes: processing the selected candidate precise utterance using a symbolic AI interpreter to generate a first candidate interpretation. In some examples, language metadata defines the relationship between a first component and a second component of a data source, wherein the fuzzy utterance identifies at least one of the first and second components, and wherein at least one candidate precise utterance identifies at least the other component of the first and second components.

[0068] At operation 420, NLQ device 120 merges the first interpretation with the second interpretation, using mixed utterance as a template for the merge and placeholder items as positions for the first interpretation within the second interpretation, thereby generating a complete interpretation (e.g., complete interpretation 252). In some examples, the merge also includes replacing the placeholder items within the second interpretation with the first interpretation to generate the complete interpretation. At operation 422, in an example, NLQ device 120 sends the complete interpretation to a query generator (e.g., query generator module 150), which is configured to generate at least one database query against a data source (e.g., database query 152) to generate query results displayed in response to a natural language query request (e.g., query result 162).

[0069] In some examples, NLQ device 120 also generates a database query against a data source, sends the database query to the data source for processing, receives the query results against the database query from the data source, and displays the query results on a user computing device (e.g., user computing device 104).

[0070] In some examples, the NLQ device 120 also identifies one or more ambiguous segments of a utterance, and based on the ambiguous segments, displays multiple explicit interpretations on a user computing device, receives a user selection for a first explicit interpretation, and, based on the user selection, modifies the mixed utterance to replace at least one ambiguous segment among the ambiguous segments with an explicit segment. Additional examples

[0071] In some examples, the hybrid NLQ processing technique described in this paper utilizes both statistical AI interpreters and symbolic AI interpreters to handle hybrid natural language queries better than some other known methods, and thus provides an improvement in the field of technology by using this combined approach.

[0072] An exemplary natural language query system includes: a processor; and a computer-readable medium storing instructions operable, when executed by the processor, to: receive a mixed utterance, the mixed utterance being a natural language query request for data from a data source; identify a first precise segment within the mixed utterance; process the first precise segment using a symbolic artificial intelligence (AI) interpreter configured to generate a first interpretation; replace the first precise segment within the mixed utterance with placeholder items to generate a first ambiguous utterance; process the first ambiguous utterance using a statistical AI interpreter configured to generate a second interpretation; merge the first interpretation and the second interpretation, the merging using the mixed utterance as a template and using placeholder items as positions for the first interpretation within the second interpretation to generate a complete interpretation; and send the complete interpretation to a query generator configured to generate at least one database query against the data source to generate query results displayed in response to the natural language query request.

[0073] Example methods include: receiving a mixed utterance, which is a natural language query request for data from a data source; identifying a first precise segment within the mixed utterance; processing the first precise segment using a symbolic artificial intelligence (AI) interpreter configured to generate a first interpretation; replacing the first precise segment within the mixed utterance with placeholder items to generate a first ambiguous utterance; processing the first ambiguous utterance using a statistical AI interpreter configured to generate a second interpretation; merging the first and second interpretations, using the mixed utterance as a template for merging and placeholder items as positions for the first interpretation within the second interpretation to generate a complete interpretation; and sending the complete interpretation to a query generator configured to generate at least one database query against the data source to generate query results displayed in response to the natural language query request.

[0074] An example computer storage device has computer-executable instructions stored thereon that, when executed by a computer, cause the computer to perform operations, including: receiving a mixed utterance, the mixed utterance being a natural language query request for data from a data source; identifying a first precise segment within the mixed utterance; processing the first precise segment using a symbolic artificial intelligence (AI) interpreter configured to generate a first interpretation; replacing the first precise segment within the mixed utterance with placeholder items to generate a first ambiguous utterance; processing the first ambiguous utterance using a statistical AI interpreter configured to generate a second interpretation; merging the first interpretation and the second interpretation, the merging using the mixed utterance as a template and placeholder items as positions for the first interpretation within the second interpretation to generate a complete interpretation; and sending the complete interpretation to a query generator configured to generate at least one database query against the data source to generate query results displayed in response to the natural language query request.

[0075] Alternatively, or in addition to the other examples described herein, examples include any combination of the following: -Receive mixed discourse; - A mixed discourse that includes at least one vague component and at least one precise component; - Mixed discourse is a natural language query request for data from a data source; -The data source is a database; - Identify precise segments within mixed discourse; -Use a symbolic artificial intelligence (AI) interpreter to process the precise segment; - A symbol AI interpreter, configured to generate interpretations; -Using placeholder entries to replace precise segments within mixed discourse, thereby generating ambiguous discourse; - Use statistical AI interpreters to process ambiguous statements; - A statistical AI interpreter, configured to generate interpretations; - Combine the first interpretation with the second interpretation; - Merging, using mixed discourse as a template for merging and using placeholder items as the position of the first interpretation within the second interpretation, thereby generating a complete interpretation; - Send a complete explanation to the query generator; - A query generator, configured to generate at least one database query against a data source to produce query results; - Display query results in response to natural language query requests; - Generate at least one database query targeting the data source; - Send at least one database query to the data source for processing; - Receive query results from the data source for at least one database query; - To display the query results on the user's computing device; - Replace the placeholder entries in the second interpretation with the first interpretation to generate a complete interpretation; - Identify one or more ambiguous segments of a discourse; - Based on one or more ambiguous segments, multiple explicit interpretations are displayed on the user's computing device; - The user's choice to receive the first explicit explanation; - Based on user selection, modify the mixed discourse to replace at least one ambiguous segment in the ambiguous segment with an explicit segment; - Submit cue words containing ambiguous utterances to a large language model (LLM); - Submit prompt words to a large language model (LLM), which includes at least one component of language metadata associated with a data source; - Submit cue words to a large language model (LLM) that includes at least one component: fuzzy utterances and language metadata associated with the data source; - Receive one or more candidate exact utterances as output from the LLM; - Utilize a symbolic AI interpreter to process one or more candidate precise utterances, thereby generating one or more candidate interpretations; - Select one of the candidate interpretations from one or more candidate interpretations as the second interpretation; - To display one or more candidate exact phrases on the user's computing device; -Receive user input indicating the selected candidate exact utterance; - The selected candidate precise utterances are processed using a symbolic AI interpreter to generate a first candidate interpretation; -Language metadata that defines the relationship between the first and second components of a data source; - Vague utterances must identify at least one of the first and second components; and - The candidate precise discourse identifies at least one component in the first and second components.

[0076] Although aspects of this disclosure have been described with reference to various examples and their associated operations, those skilled in the art will understand that combinations of operations from any number of different examples are also within the scope of aspects of this disclosure. Example operating environment

[0077] Figure 5This is a block diagram of an example computing device 500 (e.g., a computer storage device) used to implement the aspects disclosed herein, and is generally designated as computing device 500. In some examples, one or more computing devices 500 are provided for a local computing scenario. In some examples, one or more computing devices 500 are provided for a cloud computing scenario. In some examples, a combination of local and cloud computing scenarios is used. Computing device 500 is merely an example of a suitable computing environment that can be used in NLQ system 100 (e.g., as NLQ device 120) and is not intended to imply any limitation on the scope or functionality of the examples disclosed herein, whether used alone or as part of a larger set. Computing device 500 should also not be construed as having any dependencies or requirements associated with any or a combination of the components / modules shown.

[0078] The examples disclosed herein can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions such as program components that are executed by a computer or other machine, such as a personal data assistant or other handheld device. Typically, program components, including routines, programs, objects, components, data structures, etc., refer to code that performs a specific task or implements a specific abstract data type. The disclosed examples can be practiced in a variety of system configurations, including personal computers, laptops, smartphones, mobile tablets, handheld devices, consumer electronics, professional computing devices, etc. The disclosed examples can also be practiced in distributed computing environments when the task is performed by a remote processing device linked via a communication network.

[0079] Computing device 500 includes a bus 510 that directly or indirectly couples to the following devices: computer storage memory 512, one or more processors 514, one or more presentation components 516, input / output (I / O) ports 518, I / O components 520, power supply 522, and network components 524. Although computing device 500 is depicted as a single device, multiple computing devices 500 can work together and share the depicted device resources. For example, memory 512 can be distributed across multiple devices, and processors 514 can be housed with different devices.

[0080] Bus 510 indicates that it can be one or more buses (such as an address bus, a data bus, or a combination thereof). Although for clarity, Figure 5The individual boxes are shown with lines, but alternative representations can be used to depict the individual components. For example, in some examples, the presentation component, such as a display device, is an I / O component, and some examples of processors have their own memory. No distinction is made between categories such as "workstation," "server," "laptop," and "handheld device," because all these categories are envisioned in... Figure 5 Within the scope of the references to "computing device" herein, memory 512 may take the form of the computer storage media referenced below and is operatively provided for storage of computer-readable instructions, data structures, program modules, and other data for computing device 500. In some examples, memory 512 stores one or more of an operating system, a general-purpose application platform, or other program modules and program data. Therefore, memory 512 is capable of storing and accessing data 512a and instructions 512b, which can be executed by processor 514 and configured to perform the various operations disclosed herein.

[0081] In some examples, memory 512 includes computer storage media. Memory 512 may include any number of memories associated with or accessible by computing device 500. Memory 512 may be internal to computing device 500 (e.g., Figure 5 The memory 512 may be located outside the computing device 500 (not shown) or both (not shown). Alternatively or additionally, the memory 512 may be distributed across multiple computing devices 500, for example, in a virtualized environment where instruction processing is performed on multiple computing devices 500. For the purposes of this disclosure, “computer storage medium,” “computer storage memory,” “memory,” and “memory device” are synonymous terms for computer storage memory 512, and none of these terms include a carrier wave or propagation signaling.

[0082] The processors 514 may include any number of processing units that read data from various entities such as memory 512 or I / O components 520. Specifically, the processors 514 are programmed to execute computer-executable instructions for implementing aspects of this disclosure. These instructions may be executed by a processor, by multiple processors within computing device 500, or by a processor external to client computing device 500. In some examples, the processors 514 are programmed to execute instructions such as those shown in the flowcharts discussed below and depicted in the accompanying drawings. Furthermore, in some examples, the processors 514 represent implementations of analog techniques for performing the operations described herein. For example, the operations may be performed by analog client computing device 500 and / or digital client computing device 500. The presentation components 516 present data indications to a user or other device. Exemplary presentation components include: display devices, speakers, printing components, vibration components, etc. Those skilled in the art will understand and recognize that computer data can be presented in a variety of ways, such as visually in a graphical user interface (GUI), audibly through a speaker, wirelessly between computing devices 500, via a wired connection, or otherwise. I / O port 518 allows computing device 500 to be logically coupled to other devices including I / O components 520, some of which may be built-in. Example I / O components 520 include, for example, but not limited to, microphones, joysticks, game controllers, disc-shaped satellite antennas, scanners, printers, wireless devices, etc.

[0083] Computing device 500 can operate in a network environment via network component 524 using a logical connection to one or more remote computers. In some examples, network component 524 includes a network interface card and / or computer-executable instructions (e.g., a driver) for operating the network interface card. Communication between computing device 500 and other devices can occur over any wired or wireless connection using any protocol or mechanism. In some examples, network component 524 is operable to use short-range communication technologies (e.g., Near Field Communication (NFC), Bluetooth) over public, private, or hybrid (public and private) environments using transport protocols. TM Data is wirelessly transmitted between devices (such as brand communication devices, etc.) or combinations thereof. Network component 524 communicates with remote resource 528 (e.g., cloud resource) across network 530 via wireless communication link 526 and / or wired communication link 526a. Various examples of communication links 526 and 526a include wireless connections, wired connections, and / or dedicated links, and in some examples, are at least partially routed via the Internet.

[0084] Although described in conjunction with example computing device 500, the examples of this disclosure can be implemented using many other general-purpose or special-purpose computing system environments, configurations, or devices. Examples of well-known computing systems, environments, and / or configurations applicable to various aspects of this disclosure include, but are not limited to, smartphones, mobile tablets, mobile computing devices, personal computers, server computers, handheld or laptop devices, multiprocessor systems, game consoles, microprocessor-based systems, set-top boxes, programmable consumer electronics, mobile phones, mobile computing and / or communication devices in wearable or accessory form factors (e.g., watches, glasses, headsets, or headphones), network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, virtual reality (VR) devices, augmented reality (AR) devices, mixed reality devices, holographic devices, etc. Such systems or devices can accept input from users in any manner, including input devices such as keyboards or pointing devices, input via gestures, proximity input (e.g., by hovering), and / or input via voice.

[0085] Examples of this disclosure can be described in the general context of computer-executable instructions, such as program modules, that are executed by one or more computers or other devices as software, firmware, hardware, or a combination thereof. Computer-executable instructions can be organized into one or more computer-executable components or modules. Typically, program modules include, but are not limited to, routines, programs, objects, components, and data structures that perform a particular task or implement a particular abstract data type. Aspects of this disclosure can be implemented using any number and organization of such components or modules. For example, aspects of this disclosure are not limited to the specific computer-executable instructions or specific components or modules shown in the accompanying drawings and described herein. Other examples of this disclosure may include different computer-executable instructions or components having more or fewer functions than those shown and described herein. In examples involving general-purpose computers, aspects of this disclosure transform a general-purpose computer into a special-purpose computing device when configured to execute the instructions described herein.

[0086] By way of example and not limitation, computer-readable media include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable memory implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, etc. Computer storage media are tangible and mutually exclusive with communication media. Computer storage media are implemented in hardware and exclude carrier waves and propagating signals. For the purposes of this disclosure, computer storage media do not include signals themselves. Exemplary computer storage media include hard disks, flash drives, solid-state memory, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage devices, magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible to computing devices. In contrast, communication media typically embody computer-readable instructions, data structures, program modules, etc., in the form of modulated data signals such as carrier waves or other transmission mechanisms, and include any information transmission medium.

[0087] The execution or order of operations in the examples of this disclosure shown and described herein is not required and may be performed in different orders in various examples. For example, it is contemplated that a particular operation may be performed before, simultaneously with, or after another operation, or that a particular operation may be performed within the scope of aspects of this disclosure. When introducing elements of aspects of this disclosure or examples thereof, the articles “a,” “an,” “the,” and “described” are intended to indicate the presence of one or more elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that additional elements may be present in addition to those listed. The term “exemplary” is intended to mean “an example.” The phrase “one or more of the following”: “A, B, and C” means “at least one of A and / or at least one of B and / or at least one of C.”

[0088] Having described aspects of this disclosure in detail, it will be clear that modifications and variations are possible without departing from the scope of the aspects of this disclosure as defined in the appended claims. Since various changes can be made to the above-described constructions, products, and methods without departing from the scope of the aspects of this disclosure, everything included in the foregoing description and shown in the drawings should be interpreted as illustrative rather than restrictive.

Claims

1. A natural language query system (100), comprising: Processor (514); as well as A computer-readable medium stores instructions that, when executed by the processor (514), are used to: Receive (410) a mixed discourse (110), the mixed discourse being a natural language query request for data from a data source (160); The first precise segment (214) within the mixed discourse (110) is identified by (412). The first precise segment (214) is processed (414) using a symbolic artificial intelligence (AI) interpreter (132), which is configured to generate a first interpretation (220). The first precise segment (214) within the mixed discourse (110) is replaced with a placeholder item (416) to generate the first ambiguous discourse (232). The first ambiguous utterance (232) is processed (418) using a statistical AI interpreter (136), which is configured to generate a second interpretation (240). The first interpretation (220) and the second interpretation (240) are merged (420) using (i) the mixed discourse (110) as a template for the merge and (ii) the placeholder item as the position of the first interpretation (220) within the second interpretation (240) to generate a complete interpretation (252). as well as The complete explanation (252) is sent (422) to the query generator (150), which is configured to generate a database query (152) against the data source (160) to generate query results (162) displayed in response to the natural language query request.

2. The natural language query system according to claim 1, wherein the instruction further operates to: Generate the database query for the data source; The database query is sent to the data source for processing; Receive query results from the data source for the database query; as well as The query results are then displayed on the user's computing device.

3. The natural language query system according to claim 1, wherein the merging further comprises: The placeholder entries in the second interpretation are replaced with the first interpretation to generate the complete interpretation.

4. The natural language query system according to claim 1, wherein the instruction further operates to: Identify the ambiguous segments of the utterance; Based on the fuzzy segment, multiple explicit interpretations are displayed on the user's computing device; Receive user selection for the first explicit interpretation; and Based on the user's selection, the mixed discourse is modified to replace at least one ambiguous segment in the ambiguous segment with a definite segment.

5. The natural language query system according to claim 1, wherein processing the first fuzzy utterance using a statistical AI interpreter further includes: Submit prompt words to the Large Language Model (LLM), the prompt words comprising the first fuzzy utterance and components of language metadata associated with the data source; Receive candidate precise utterances from the LLM; The candidate precise utterances are processed using the symbol AI interpreter to generate candidate interpretations; as well as One of the candidate interpretations is selected as the second interpretation.

6. The natural language query system according to claim 5, wherein processing the first fuzzy utterance using a statistical AI interpreter further includes: The candidate precise utterances are displayed on the user's computing device; as well as Receive user input indicating the selected candidate exact utterance. The process of processing the candidate precise discourse includes: using the symbol AI interpreter to process the selected candidate precise discourse, thereby generating a first candidate interpretation.

7. The natural language query system of claim 5, wherein the language metadata defines the relationship between a first component and a second component of the data source, wherein the fuzzy utterance identifies at least one of the first component and the second component, and wherein at least one of the candidate precise utterances identifies at least the other of the first component and the second component.

8. A computer-implemented method (400), comprising: Receive (410) a mixed discourse (110), the mixed discourse (110) being a natural language query request for data from a data source (160); The first precise segment (214) within the mixed discourse (110) is identified by (412). The first precise segment (214) is processed (414) using a symbolic artificial intelligence (AI) interpreter (132), which is configured to generate a first interpretation (220). The first precise segment (214) within the mixed discourse (110) is replaced with a placeholder item (416) to generate the first ambiguous discourse (232). The first ambiguous utterance (232) is processed (418) using a statistical AI interpreter (136), which is configured to generate a second interpretation (240). The first interpretation (220) and the second interpretation (240) are merged (420), the merging using the mixed discourse (110) as a template for the merging and the placeholder item as the position of the first interpretation (220) within the second interpretation (240), thereby generating a complete interpretation (252). as well as The full explanation (252) is sent (422) to the query generator (50), which is configured to generate a database query (152) against the data source (160) to generate query results (162) displayed in response to the natural language query request.

9. The computer-implemented method according to claim 8, further comprising: Generate the database query for the data source; The database query is sent to the data source for processing; Receive query results from the data source for the database query; as well as The query results are then displayed on the user's computing device.

10. The computer-implemented method of claim 8, wherein the merging further comprises: The placeholder entries in the second interpretation are replaced with the first interpretation to generate the complete interpretation.

11. The computer-implemented method according to claim 8, further comprising: Identify the ambiguous segments of the utterance; Based on the fuzzy segment, a clear explanation is displayed on the user's computing device; Receive the user's choice regarding the first explicit interpretation; as well as Based on the user's selection, the mixed discourse is modified to replace at least one ambiguous segment in the ambiguous segment with a definite segment.

12. The computer-implemented method according to claim 8, wherein processing the first ambiguous utterance using a statistical AI interpreter further includes: Submit prompt words to the Large Language Model (LLM), the prompt words comprising the first fuzzy utterance and components of language metadata associated with the data source; Receive candidate precise utterances from the LLM; The candidate precise utterances are processed using the symbol AI interpreter to generate candidate interpretations; as well as One of the candidate interpretations is selected as the second interpretation.

13. The computer-implemented method of claim 12, wherein processing the first ambiguous utterance using a statistical AI interpreter further comprises: The candidate precise utterances are displayed on the user's computing device; as well as Receive user input indicating the selected candidate exact utterance. The process of processing the candidate precise discourse includes: using the symbol AI interpreter to process the selected candidate precise discourse, thereby generating a first candidate interpretation.

14. The computer-implemented method of claim 12, wherein the language metadata defines the relationship between a first component and a second component of the data source, wherein the fuzzy utterance identifies at least one of the first component and the second component, and wherein at least one of the candidate precise utterances identifies at least the other component of the first component and the second component.

15. A computer storage device (512) storing computer-executable instructions (512b) thereon, the computer-executable instructions causing the computer to perform operations when executed by a computer, including: Receive (410) a mixed discourse (110), the mixed discourse (110) being a natural language query request for data from a data source (160); The first precise segment (214) within the mixed discourse (110) is identified by (412). The first precise segment (214) is processed (414) using a symbolic artificial intelligence (AI) interpreter (132), which is configured to generate a first interpretation (220). The first precise segment (214) within the mixed discourse (110) is replaced with a placeholder item (416) to generate the first ambiguous discourse (232). The first ambiguous utterance (232) is processed (418) using a statistical AI interpreter (136), which is configured to generate a second interpretation (240). The first interpretation (220) and the second interpretation (240) are merged (420), the merging using the mixed discourse (110) as a template for the merging and the placeholder item as the position of the first interpretation (220) within the second interpretation (240), thereby generating a complete interpretation (252). as well as The complete explanation (252) is sent (422) to the query generator (150), which is configured to generate a database query (152) against the data source (160) to generate query results (162) displayed in response to the natural language query request.

16. The computer storage device according to claim 15, further comprising: Generate the database query for the data source; The database query is sent to the data source for processing; Receive query results from the data source for the database query; as well as The query results are then displayed on the user's computing device.

17. The computer storage device of claim 15, wherein the merging further comprises: The placeholder entries in the second interpretation are replaced with the first interpretation to generate the complete interpretation.

18. The computer storage device according to claim 15, further comprising: Identify the ambiguous segments of the utterance; Based on the fuzzy segment, a clear explanation is displayed on the user's computing device; Receive the user's choice regarding the first explicit interpretation; as well as Based on the user's selection, the mixed discourse is modified to replace at least one ambiguous segment in the ambiguous segment with a definite segment.

19. The computer storage device of claim 15, wherein processing the first ambiguous utterance using a statistical AI interpreter further comprises: Submit prompt words to the Large Language Model (LLM), the prompt words comprising the first fuzzy utterance and components of language metadata associated with the data source; Receive candidate precise utterances from the LLM; The candidate precise utterances are processed using the symbol AI interpreter to generate candidate interpretations; as well as One of the candidate interpretations is selected as the second interpretation.

20. The computer storage device of claim 19, wherein processing the first fuzzy utterance using a statistical AI interpreter further comprises: The candidate precise utterances are displayed on the user's computing device; as well as Receive user input indicating the selected candidate exact utterance. The process of processing the candidate precise discourse includes: using the symbol AI interpreter to process the selected candidate precise discourse, thereby generating a first candidate interpretation.

Images