Machine learning-augmented knowledgebase queries

Abstract

Systems, methods, and computer program products are described for employing a large language model (LLM) to generate a logic programming language query to be executed against a knowledgebase. A received query is used to determine information relevant to the query. The query and the information relevant to the query are provided to the LLM to generate the logic programming language query. The logic programming language query is executed against the knowledgebase to generate results.

Description

MACHINE LEARNING-AUGMENTED KNOWLEDGEBASE QUERIESInventors : Andreas Werner Robert PfadlerMarta Enciso Hongliang Liu Max Penveme Matthew Edward Sparrowhawk Romeo Radman ShengLei Cao Andrei Groznyi Xiaodong Zhou Nicky Bayat Suan Chin Yeo Ryan BradyBACKGROUND

[0001] Large language models (LLMs) are sophisticated artificial intelligence systems that have been trained on vast amounts of text data. LLMs have the ability to understand and generate human-like text, making them valuable for a wide range of applications, including, but not limited to, knowledgebase question-and-answer (Q&A) systems.SUMMARY

[0002] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0003] Systems, methods, and computer program products are described herein for employing a large language model (LLM) to generate a logic programming language query to be executed against a knowledgebase. A received query is used to determine information relevant to the query. The query and the information relevant to the query are provided to the LLM to generate the logic programming language query. The logic programming language query is executed against the knowledgebase to generate results.

[0004] Further features and advantages of the embodiments, as well as the structure and operation of various embodiments, are described in detail below with reference to the accompanying drawings. It is noted that the claimed subject matter is not limited to the specific embodiments described herein. Such embodiments are presented herein for illustrative purposes. Additional embodiments will be apparent to persons skilled in the relevant art(s) based on the teachings contained herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments of the present application and, together with the description, further serve to explain the principles of the embodiments and to enable a person skilled in the pertinent art to make and use the embodiments. Dashed portions of the drawings may represent optional steps and / or elements.

[0006] FIG. 1 shows a block diagram of an example system for executing LLM-generated logic programming language queries, in accordance with an embodiment.

[0007] FIG. 2 shows a block diagram of an example system for executing LLM-generated logic programming language queries, in accordance with an embodiment.

[0008] FIG. 3 depicts a flowchart of a method for executing LLM-generated logic programming language queries, in accordance with an embodiment.

[0009] FIG. 4 depicts a flowchart of a method for semantic searching for relevant information, in accordance with an embodiment.

[0010] FIG. 5 depicts a flowchart of a method for employing an LLM to generate a logic programming language query, in accordance with an embodiment.

[0011] FIG. 6 shows a block diagram of an example system for automatic knowledgebase generation, in accordance with an embodiment.

[0012] FIGS. 7A-7C depict flowcharts of methods for automatically generating a knowledgebase from various data sources, in accordance with an embodiment.

[0013] FIG. 8 shows a block diagram of an example computer system in which embodiments may be implemented.

[0014] The features and advantages of the embodiments described herein will become more apparent from the detailed description set forth below when taken in conjunction withthe drawings, in which like reference characters identify corresponding elements throughout. In the drawings, like reference numbers generally indicate identical, functionally similar, and / or structurally similar elements. The drawing in which an element first appears is indicated by the leftmost digit(s) in the corresponding reference number.DETAILED DESCRIPTIONI. Introduction

[0015] The following detailed description discloses numerous example embodiments. The scope of the present patent application is not limited to the disclosed embodiments, but also encompasses combinations of the disclosed embodiments, as well as modifications to the disclosed embodiments.

[0016] References in the specification to "one embodiment," "an embodiment," "an example embodiment," etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[0017] In the discussion, unless otherwise stated, adjectives such as “substantially” and “about” modifying a condition or relationship characteristic of a feature or features of an embodiment of the disclosure, are understood to mean that the condition or characteristic is defined to within tolerances that are acceptable for operation of the embodiment for an application for which it is intended.

[0018] The example embodiments described herein are provided for illustrative purposes and are not limiting. Further structural and operational embodiments, including modifications / alterations, will become apparent to persons skilled in the relevant art(s) from the teachings herein.

[0019] Numerous exemplary embodiments are described as follows. It is noted that any section / subsection headings provided herein are not intended to be limiting. Embodimentsare described throughout this document, and any type of embodiment may be included under any section / subsection. Furthermore, embodiments disclosed in any section may be combined with any other embodiments described in the same section and / or a different section.

[0020] As used herein, the terms “large language model” and “LLM” are used interchangeably, and are defined as an artificial neural network that is trained using a large training dataset and is capable of performing natural language processing tasks to understand user queries.II. Example Embodiments

[0021] As discussed in the Background section above, LLMs are trained using vast amounts of text data, including, but not limited to, books, articles, websites, and the like. LLMs are trained by iteratively generating models that predict the next word in a sentence given the context of the previous words. By training on a massive amount of data, the model learns grammar, syntax, factual information, and other patterns in language. To predict the next word, the model utilizes a self-attention mechanism, which allows it to capture dependencies between different words in the sequence. This mechanism helps the model to understand the context and meaning of the text.

[0022] Although LLMs can generate coherent and contextually relevant responses, they often lack a deep understanding of the context. LLMs primarily rely on statistical patterns in the training data and might generate plausible sounding but incorrect or nonsensical answers. This LLM phenomenon is also known as “hallucination.” Furthermore, LLMs struggle to generate accurate responses that require information that is not included in the training data. As such, LLMs have limited utility when responding to questions about recent events and / or proprietary information.

[0023] Training an LLM requires powerful hardware, such as GPUs (Graphics Processing Units), and significant computational time to process the vast amounts of data and optimize the model's parameters. Given these computational costs, it is infeasible to constantly retrain LLMs with new training data that include the latest information. Additionally, most companies do not have the resources to train an LLM based on their own proprietary data.

[0024] Attempts to overcome the shortcomings of LLMs include retrieval augmented generation. In retrieval augmented generation, a user query is first used to retrieve relevantdata, and then a prompt is generated to request the LLM to answer the user query based on the retrieved data. The retrieved data provide additional contextual information that improves the LLM’s response by focusing the response to information contained in the retrieved data. Furthermore, proprietary information may be included in the retrieved data to augment the knowledge of the LLM, allowing the LLM to respond to queries concerning proprietary information. While the additional context provided by the retrieved data improve the accuracy of the responses, it does not completely eliminate the possibility of incorrect or false information. Ultimately, the LLM is still generating a response based on statistical patterns in text data.

[0025] The embodiments described herein are directed techniques for using an LLM to generate logic programming language queries to be executed against knowledgebase Q&A systems. For example, the knowledgebase Q&A systems may receive a user query in a natural language. In embodiments, the knowledgebase Q&A systems may determine an embedding vector for the user query and perform a semantic search of the knowledgebase for information relevant to the user query. The knowledgebase Q&A systems may then provide the relevant information in a prompt to an LLM requesting the LLM to convert the user query to a logic programming language syntax based on the relevant information. The response from the LLM may include a logic programming language query that may be executed against the knowledgebase. The knowledgebase Q&A systems may then return the results from executing the logic programming language query to the user. In embodiments, the knowledgebase Q&A systems may format, translate, and / or otherwise manipulate the results prior to returning them to the user.

[0026] Knowledgebase Q&A systems implemented using the techniques disclosed herein leverage the LLM to convert the natural language user query into a logic programming language query that is executed against the knowledgebase. By employing the LLM to convert the natural language user query into a logic programming language query, the knowledgebase Q&A systems do not require the user to have any prior knowledge of the logic programming language syntax or the knowledgebase schema, and provides the user increased flexibility in how they search the knowledgebase.

[0027] These and further embodiments are disclosed herein that enable the functionality described above and further such functionality. Such embodiments are described in further detail as follows.

[0028] For instance, FIG. 1 shows a block diagram of an example system 100 for executing LLM-generated logic programming language queries against a knowledgebase. As shown in FIG. 1, system 100 may include one or more clients 102, one or more servers 104, one or more networks 106, a knowledgebase 108, and a large language model (LLM) 110. In embodiments, client(s) 102 may further include a user interface (UI) 112 that may be used to interact with server(s) 104 via network(s) 106. Furthermore, server(s) 104 may include a user interface (UI) manager 114 and a Q&A system 116. Each of the components of system 100 are described in detail as follows.

[0029] Client(s) 102 include any computing device suitable for performing functions that are ascribed thereto in the following description, as will be appreciated by persons skilled in the relevant art(s), including those mentioned elsewhere herein or otherwise known. Various example implementations of client(s) 102 are described below in reference to computing device 802 of FIG. 8. Client(s) 102 are communicatively connected to server(s) 104 through network(s) 106. In embodiments, client(s) 102 may include a user interface 112 that may be used by a user to interact with server(s) 104.

[0030] UI 112 may include any interface for performing functions that are ascribed thereto in the following description, as will be appreciated by persons skilled in the relevant art(s), including those mentioned elsewhere herein or otherwise known. UI 112 may include, but are not limited to, software, firmware, and / or hardware components that allow a user to interact with server(s) 104 and / or components thereof using one or more modalities, such as, but not limited to, a graphical user interface (GUI), a voice-based interface, a voice- over-IP (VoIP) interface, a text-based interface, a short message service (SMS) interface, a chatbot interface, a chat interface, an instant messaging (IM) interface, a teleconference interface, a touch-based interface, a touchscreen interface, a camera-based interface, a sign language interface, a telepathic interface, and / or a multi-modal interface that permits the user to concurrently and / or serially employ one or more of the aforementioned modalities to interact with server(s) 104 and / or components thereof.

[0031] Network(s) 106 may include any one or more networks suitable for performing functions that are ascribed thereto in the following description, as will be appreciated by persons skilled in the relevant art(s), including those mentioned elsewhere herein or otherwise known. Various example implementations of network(s) 106 are described below in reference to network 804 of FIG. 8.

[0032] Server(s) 104 may include one or more of any server computing device suitable for performing functions that are ascribed thereto in the following description, as will be appreciated by persons skilled in the relevant art(s), including those mentioned elsewhere herein or otherwise known. Server(s) 104 may be implemented separately, or with reference to the exemplary computing environment 800 of FIG. 8 described below, server(s) 104 may alternatively be implemented as on-premises servers 892, and / or as part of network-based server infrastructure 870. In embodiments, server(s) 104 may include UI manager 114 and Q&A system 116.

[0033] UI manager 114 may be configured to receive input from the user via UI 112 and provide output to the user via UI 112. UI manager 114 may also generate and provide UI 112 to client(s) 102. UI manager 114 may provide input received from client(s) 102 to Q&A system 116 and provide responses from Q&A system 116 to chent(s) 102. In embodiments, UI manager 114 may pre-process input received from client(s) 102 prior to providing the input to Q&A system 116. For example, UI manager 114 may include a speech recognition engine (not shown) configured to convert voice input from client(s) 102 into text that is provided to Q&A system 116. In embodiments, UI manager 114 may also include a text-to-speech (TTS) engine (not shown) configured to convert a text response from Q&A system 116 into synthesized speech that is provided to client(s) 102.

[0034] Q&A system 116 may be configured to receive user queries from client(s) 102 via UI manager 114 and provide responses to the user queries to client(s) 102 via UI manager 114. In embodiments, Q&A system 116 may receive a user query' as text data and may perform a semantic search of knowledgebase 108 using the user query to retrieve information from knowledgebase 108 that may be relevant to the user query'. Q&A system 116 may then provide the retrieved information to LLM 110 in a prompt requests LLM 110 to convert the user query into a logic programming language syntax based on the retrieved information. In response to the prompt, Q&A system 116 may receive a logic programming language query that may be executed against knowledgebase 108. Q&A system 116 may then execute the logic programming language query against knowledgebase 108 and receive results from knowledgebase 108. In embodiments, Q&A system 116 may provide the results from knowledgebase 108 to client(s) 102 via UI manager 1 14 as a response. In embodiments, Q&A system 116 may reformat, translate, and / or otherwise manipulate theresults from knowledgebase 108 prior to providing the results to client(s) 102. Q&A system 116 will be described in greater detail in conjunction with FIG. 2 below.

[0035] Knowledgebase 108 may include repositories of information that store and organize knowledge on specific subjects and / or domains. In embodiments, knowledgebase 108 may store, and / or retrieve knowledge in a structured manner. For instance, knowledgebase 108 may be structured using one or more formats, such as, but not limited to, logic programming languages (e.g., Prolog, Datalog, Cypher. PL, Loglisp, and / or derivatives thereof), knowledge graphs, and / or relational databases. In embodiments, knowledgebase 108 may include facts, concepts, relationships, rules, and / or other relevant data. For instance, knowledgebase 108 may include facts, concepts, relationships, rules as one or more logical statements or constructs. The use of logic programming languages enables queries to support recursion, logical inferences, and / or compositionality, some of which may not be supported by SQL-based queries. In embodiments, logic programming languages may be supported by relational backend for scalability. For instance, the logical statements or constructs may be stored in a relational database (e g., SQL database, etc.) in order to improve scaling of very large knowledgebases.

[0036] In embodiments, knowledgebase 108 may be domain-specific and include facts, concepts, relationships, rules, and / or other relevant data about a specific subject or domain. For instance, a pharmaceutical-specific knowledgebase may include, but is not limited to, facts, concepts, relationships, rules, and / or other relevant data relating to one or more drugs, generic drug names, brand names, drug manufacturers, pharmacies, drug indications, side effects, dosage information, treatments, drug interactions, clinical trials, co-development agreements, trial outcomes, drug approvals, drug development stage, and / or the like. In embodiments, knowledgebase 108 may include proprietary or internal information. For instance, propnetary or internal knowledgebase may include, but is not limited to, facts, concepts, relationships, rules, and / or other relevant data relating to an entity (e.g., company, organization, group, etc.) such as, but not limited to, trade secrets, source code, product recipes or formulas, process information, sales numbers, employee data, production information, test results, protein sequences, internal data, market research, and / or the like.

[0037] In embodiments, information in knowledgebase 108 may include predicates (e g., facts, relationships, rules, etc.) that may be manually, automatically, and / or semi- automatically entered into knowledgebase 108. Furthermore, predicates can beautomatically generated using plugins or hooks to invoke external code (e.g., an inference engine, machine learning model, inference model, etc.) to predict or infer relationships from other information, such as, but not limited to, knowledgebase 108, the user query, external sources, and / or the like.

[0038] In embodiments, search logic and / or semantics may be customized by adjusting or modifying predicates (e.g., facts, relationships, rules, etc.) in knowledgebase 108. In embodiments, customized search logic and / or semantics may include, but are not limited to, Boolean searches, wildcard searches, and / or the like. Such customizations simplify modification of search logic and / or semantics when compared to text2sql or text-only retrieval based systems. Furthermore, a feedback loop may be employed to manually, automatically, and / or dynamically improve the predicates (e.g., facts, relationships, rules, etc.) in knowledgebase 108. In embodiments, the feedback may include, but is not limited to, internal feedback, external feedback, a combination of internal and external feedback, and / or the like. The incorporated feedback loop may improve the accuracy and / or relevancy of future responses.

[0039] LLM 110 may include, but are not limited to, publicly available LLMs (e.g., GPT- 3, GPT-4, PaLM2, etc ), fine-tuned LLMs, private LLMs, and / or the like. In embodiments, LLM 110 may be trained using vast amounts of training data, including, but not limited to, publicly available data (e.g., books, articles, websites, databases, etc.), and / or proprietary or private data. The training data may, in embodiments, include information regarding programming languages, such as, but not limited to, logic programming languages (e.g., Prolog, Datalog, etc.), object-oriented programming languages (e.g., Java, C++, etc.). In embodiments, LLM 110 may receive a prompt from Q&A system 116 that includes a user query and information related to the user query, and that requests conversion of the user query into a logic programming language syntax based on the information related to the user query. In embodiments, the information related to the user query may include facts, concepts, relationships, rules, and / or other relevant data from knowledgebase 108 that allow LLM 110 to infer a schema of knowledgebase 108, and to convert the user query into a logic programming language query. LLM may then return the generated logic programming language query to Q&A system 1 16. In embodiments, the training data may include, but are not limited to, specific data such as drugs, companies, mechanism of actions, clinical trials, deals, patents, drug safety data / intelligence, biomarkers, real worlddata, indications, conditions, therapy areas, chemical structures, regulatory designations, milestones / catalysts, regulatory, commercialization and launch, forecasts, financial information about the companies developing drugs, among others without departing from the meaning and / or scope of the described embodiments. Embodiments may be implemented and across various product suites and services offered by Clarivate™ and / or its associated entities, without any limitations and without departing from the meaning and / or scope of the described embodiments. In embodiments, LLM 110 may be implemented using an LLM without retraining when the LLM is sufficiently powerful.

[0040] Embodiments described herein may operate in various ways to respond to a user query by executing an LLM-generated logic programming language query against a knowledgebase. For instance, FIG. 2 shows a block diagram of an example system 200 for executing LLM-generated logic programming language queries, in accordance with an embodiment. System 200 may include client(s) 102, server(s) 104, network(s) 106, knowledgebase 108, LLM 110, UI 112, UI manager 114, and Q&A system 116, as shown and described in conjunction with FIG. 1 above. As shown in FIG. 2, Q&A system 116 may further include a query processor 202, a knowledgebase interfacer 204, an LLM interfacer 206, and a response generator 208. Furthermore, query processor 202 may further include an embedding vectorizer 210, knowledgebase interfacer 204 may further include a semantic searcher 212, and a knowledgebase querier 214, and LLM interfacer 206 may further include a prompt generator 216, and an LLM querier 218. Each of the components of system 200 are described in detail as follows.

[0041] Query processor 202 may receive a user query from client(s) 102 via UI manager 114. For example, query processor 202 may receive the following user queries.Example Query 1: " I n which count rie s ha s Ability been launched for treatment of depre s s ion or similar indications ?"Example Query 2: " Generate a mar ket report for Ability . "In embodiments, query processor 202 may process user queries to determine a task (e.g., generating a report, analyzing data, retrieve data, etc.) and / or data required to perform the task. Query processor 202 may employ embedding vectorizer 210 to generate embeddingvectors representative of the user queries. Furthermore, query processor 202 may provide the user query, the determined task, and / or the determined data required to perform the task to one or more of knowledgebase interfacer 204, LLM interfacer 206, and / or response generator 208.

[0042] In embodiments, query processor 202 may support multi-step user interactions by maintaining recent user queries. For example, query processor 202 may maintain, for at least some users, a historical record of user queries. In embodiments, the historical record of user queries may, for example, be maintained at client(s) 102 and / or server(s) 104 as a user profile, browser cookie, and / or the like. A historical record of the user queries provide contextual information that allow query processor to piece together a user’s intent from a plurality of user queries received over a period of time. In embodiments, query processor 202 may further generate, in response to a first question from a user, follow-up question seeking clarification related to the first question. Query processor 202 may present the follow-up question to the user via UI 112, and receive a response to the follow-up question from the user via UI 112. Query processor 202 may combine information from the first question and the user’s response to the follow-up question to generate a combined user query. In embodiments, query processor 202 may perform the functions discussed above based on the combined user query.

[0043] Embedding vectorizer 210 may include, but are not limited to, models or neural networks to generate numerical representations of words, sentences, and / or paragraphs. In embodiments, embedding vectorizer 210 may include models trained using machine learning algorithms, such as, but not limited to, word2vec and / or GloVe (Global Vectors for Word Representation). Embedding vectors generated by embedding vectorizer 210 may capture semantic relationships and contextual information of the user queries, enabling Q&A system 116 to understand the meaning of the user queries. In embodiments, embedding vectorizer 210 may provide the generated embedding vectors to knowledgebase interfacer 204.

[0044] In embodiments, embedding vectorizer 210 may operate according to various implementations. For instance, in one implementation embedding vectorizer 210 may employ Word2Vec, which uses neural networks to learn word embeddings. Word2Vec utilizes either the Continuous Bag-of-Words (CBOW) or Skip-gram model. The CBOW model predicts a target word based on its surrounding context words, while the Skip-grammodel predicts context words given a target word. These models are trained on large amounts of text data, and the weights of the neural network are adjusted through backpropagation to optimize the prediction task. The resulting weights of the hidden layer, which represent the embedding vectors, capture the semantic meaning and relationships between words. In another implementation, embedding vectorizer may employ GloVe, which combines co-occurrence statistics with matrix factorization techniques to generate word embeddings. GloVe constructs a global word co-occurrence matrix and then factorizes it to obtain the word vectors. These techniques enable the generation of highdimensional embedding vectors that encode semantic information and facilitate various natural language processing tasks. In embodiments, embedding vectorizer 210 may employ transformer-based embedding models to generate the embedding vectors, without departing from the meaning and scope of described embodiments.

[0045] Knowledgebase interfacer 204 may interact with knowledgebase 108 to retrieve information (e.g., facts, concepts, relationships, rules, etc.) that are relevant to the user query, and to execute logic programming language queries to retrieve data to respond to the query. Knowledgebase interfacer 204 may receive an embedding vector representative of a user query from query processor 202 and employ semantic searcher 212 to search knowledgebase 108 for information (e.g., facts, concepts, relationships, rules, etc.) that are semantically similar to the embedding vector. Knowledgebase interfacer 204 may provide the relevant information retrieved from knowledgebase 108 to LLM interfacer 206.

[0046] Semantic searcher 212 may determine a similarity between a first embedding vector and a plurality of indexed embedding vectors and retrieve the information corresponding to the indexed embedding vectors having the highest similarity to the first embedding vector from knowledgebase 108. In embodiments, sematic searcher 212 may determine similarities between the first embedding vector and the plurality of indexed embedding vectors by calculating metrics such as, but not limited to, cosine distance and / or Euclidean distance. For instance given a user query, the embedding vector of the query is compared to embedding vectors representative of the contents (e.g., facts, concepts, relationships, rules, etc.) of knowledgebase 108, and the embedding vectors with the highest similarity scores (e.g., shortest, or smallest cosine and / or Euclidean distances) are considered the most semantically relevant. Semantic searching using embedding vectors enables the retrieval of results that are conceptually related or contextually similar, facilitating more accurate andcontextualized information retrieval. In embodiments, the plurality of indexed embedding vectors representative of the contents of knowledgebase 108 may be determined or calculated prior to receipt of the user query and stored at server(s) 104, knowledgebase 108, and / or any other location accessible to semantic searcher 212.

[0047] Semantic searcher 212 may receive information relevant to the user query (e.g., facts, concepts, relationships, rules, etc.) from knowledgebase 108 and provide the received relevant information to LLM interfacer 206. Returning to the example queries above, Semantic searcher 212 may receive the following relevant facts, concepts, relationships, and / or rules from knowledgebase 108 in response to a semantic search based an embedding vector representative of example query 1 and / or example query 2.1: "f ind_drug ( Drug , String ) synonym ( Drug, String ) ; synonym ( Strin , Drug ) ; ( dru ( Drug) , Drug=String) "2: synonym ( ' aripiprazole , ' Abilif y ' , drug )

[0048] LLM interfacer 206 may receive a user query from query processor 202 and / or knowledgebase interfacer 204, and information relevant to the user query from knowledgebase interfacer 204. In embodiments, LLM interfacer may employ prompt generator 216 to generate a prompt based on the received user query and information relevant to the user query, and employ LLM querier 218 to interface with LLM 110 to convert the user query into a logic programming language query based on the information relevant to the user query.

[0049] Prompt generator 216 may generate a prompt to request LLM 110 to convert the user query into a logic programming language query. For instance, returning to the example user queries above, prompt generator 216 may generate and provide the following prompt to LLM Querier 218.Here are some related facts in PROLOG syntax :"f ind_drug ( Drug , String ) synonym ( Drug, String ) ; synonym ( String , Drug ) ; ( drug ( Drug) , Drug=String) " synonym ( ' aripiprazole , ' Abilif y ' , drug) .Now please convert the query "In which countries has Abilify been launched for treatment of depression or similar indications ?" into a PROLOG syntax .

[0050] LLM querier 218 may receive the generated prompt from prompt generator 216 and query LLM 110 using the generated prompt. In response, LLM querier 218 may receive, from LLM 110, a logic programming language query that may be executed against knowledgebase 108. For instance, returning to the example user queries above, LLM querier 218 may receive the following response from LLM 110.( synonym ( 'Abilify' , Drug , drug) ; synonym ( Drug , 'Ability, drug ) ; ( Drug=' Abilify) ) , developing ( Company, Drug , Indicati on , Developmentstatus , Country) , marketed_status ( Developme ntaStatus ) , similar ( ' Depression' , Indication , indication, K) , K<10 .In embodiments, LLM querier 218 may provide the received logic programming language query to knowledgebase querier 214.

[0051] Knowledgebase querier 214 may receive a logic programming language query from LLM interfacer 206 and execute the logic programming language query against knowledgebase 108 to retrieve information (e.g., facts, concepts, relationships, rules, etc.) that may be used to respond to the user query. Knowledgebase querier 214 may provide the retrieved information from knowledgebase 108 to response generator 208.

[0052] Response generator 208 may receive a task from query processor 202, and / or data required to perform the task from knowledgebase interfacer 204. Response generator 208 may perform the received task based on the received data to generate a response to the user query. In embodiments, the received task may include the retrieval of data, and response generator 208 generates a response by formatting the received data in a format suitable for transmission to, and / or output by, client(s) 102. In other embodiments, the received task may require further processing of the received data (e.g., report generation, trend analysis, predictive modeling, etc.), and response generator 208 may process the received data using one or more sub-components (not shown) or external components (not shown) to generatea response to the user query. For instance, response generator 208 may employ a report generator (not shown) to generate reports that summarize the received data in one or more formats (e.g., text, images, graphs, charts, etc.). Furthermore, response generator 208 may employ an analysis and / or prediction model (not shown) to generate trends and / or predictions (e.g., growth trends, sales trends, market trends, predicted growth, etc.) based on the received data. Response generator 208 may provide the generated response to client(s) 102 via UI manager 114.

[0053] In embodiments, response generator 208 may analyze the received data to discover patterns, relationships, trends, and / or any other insights from the received data. For example, response generator 208 may, in embodiments, analyze lab results in the received data to determine trends, causal relationships, and / or side effects of a drug during a clinical trial. Furthermore, in embodiments, response generator 208 may generate content, including, but not limited to, reports, graphs, charts, tables, infographics, and / or the like to convey the patterns, relationships, trends, and / or any other insights to the user. Additionally, in embodiments, response generator 208 may interact with UI manager 114 to provide the generated content to chent(s) 102 through one or more channels, including, but not limited to, mobile apps, website, webpage, SMS, Slack, Salesforce, Facebook, Veeva, Microsoft® Teams®, and / or the like.

[0054] Since response generator 208 generates responses based on data retrieved by querying knowledgebase 108, the responses do not contain false or incorrect information. Unlike previous attempts at overcoming LLM hallucinations, embodiments disclosed herein greatly reduce LLM hallucinations. Furthermore, any hallucinations by LLM 110 will not affect the factuality and / or correctness of the responses generated by response generator 208 because LLM 110 is only employed to generate a logic programming language query that is used to retrieve data directly from knowledgebase 108. Additionally, responses generated by response generator 208 can readily be verified for accuracy by accessing and comparing the response to data in knowledgebase 108.

[0055] Embodiments described herein may operate in various ways to respond to a user query of a knowledgebase. For instance, FIG. 3 depicts a flowchart 300 of a method for executing LLM-generated logic programming language queries, in accordance with an embodiment. Server(s) 104, UI manager 114, Q&A system 116, query processor 202, knowledgebase interfacer 204, LLM interfacer 206, response generator 208, embeddingvectorizer 210, sematic searcher 212, knowledgebase querier 214, prompt generator 216, LLM querier218 of FIGS. 1 and / or 2 may operate according to flowchart 300, for example. Note that not all steps of flowchart 300 may need to be performed in all embodiments, and in some embodiments, the steps of flowchart 300 may be performed in different orders than shown. Flowchart 300 is described as follows with respect to FIGS. 1 and 2 for illustrative purposes.

[0056] Flowchart 300 starts at step 302. In step 302, a first query is received. For example, query processor 202 of Q&A system 116 may receive a user query from client(s) 102 via UI manager 114. As described above, UI manager 114 may pre-process a user query received from client(s) 102 prior to providing the user query to query processor 202 of Q&A system 116. For example, UI manager 114 may include a speech recognition engine (not shown) configured to convert voice input from client(s) 102 into text that is provided to query processor 202 of Q&A system 116.

[0057] In step 304, information relevant to the first query is determined. For example, semantic searcher 212 of Q&A system 116 may retrieve, from knowledgebase 108, information that are semantically similar to the first query. In embodiments, semantic searcher 212 may provide the retrieved information to LLM interfacer 206. The operations of step 304 will be discussed in greater detail in conjunction with FIG. 4 below.

[0058] In step 306, a large language model is used to generate a logic programming language query based at least on the first query and the information relevant to the first query. For example, LLM interfacer 206 of Q&A system 116 may interact with LLM 110 to convert the user query into a logic programming language query based on information (e.g., facts, concepts, relationships, rules, etc.) relevant to the first query retrieved from knowledgebase 108. In embodiments, LLM interfacer 206 may provide the logic programming language query to knowledgebase interfacer 204 for execution against knowledgebase 108. The operations of step 306 will be discussed in greater detail in conjunction with FIG. 5 below.

[0059] In step 308, the logic programming language query is executed against a knowledgebase to generate results. For example, knowledgebase querier 214 of knowledgebase interfacer 204 may execute the logic programming language query generated by LLM 110 against knowledgebase 108 and receive results from the query fromknowledgebase 108. In embodiments, knowledgebase querier 214 may provide the results to response generator 208.

[0060] In step 310, the results are returned as a response to the first query. For example, response generator 208 may generate a response based on the results received from knowledgebase querier 214 and provide the generated response to client(s) 102 via UI manager 114. As discussed above, in embodiments, response generator 208 may generate a response by formatting the results in a format suitable for transmission to, and / or output by, client(s) 102. In other embodiments, response generator 208 may process the received data using one or more sub-components (not shown) or external components (not shown) to generate a response to the user query. For instance, response generator 208 may employ a report generator (not shown) to generate reports that summarize the received data in one or more formats (e.g., text, images, graphs, charts, etc.) or employ an analysis and / or prediction model (not shown) to generate trends and / or predictions (e.g., growth trends, market trends, predicted growth, etc.) based on the received data.

[0061] Embodiments described herein may operate in various ways to semantically search a knowledgebase for information relevant to a user query. For instance, FIG. 4 depicts a flowchart 400 of a method for semantic searching for relevant information, in accordance with an embodiment. Server(s) 104, query processor 202, knowledgebase interfacer 204, embedding vectorizer 210, sematic searcher 212 of FIGS. 1 and / or 2 may operate according to flowchart 400, for example. Note that not all steps of flowchart 400 may need to be performed in all embodiments, and in some embodiments, the steps of flowchart 400 may be performed in different orders than shown. Flowchart 400 is described as follows with respect to FIGS. 1 and 2 for illustrative purposes.

[0062] Flowchart 400 starts at step 402. In step 402, a first embedding vector is determined based on the first query. For example, embedding vectorizer 210 may generate or otherwise determine an embedding vector representative of the user query. As discussed above, embedding vectorizer 210 may generate a numerical representation (e.g., vector) of the user query using models trained based on machine learning algorithms, such as, but not limited to, word2vec and / or GloVe. In embodiments, embedding vectorizer 210 may provide the generated embedding vectors to knowledgebase interfacer 204.

[0063] In step 404, a semantic search of the knowledgebase is performed using the first embedding vector. For example, semantic searcher 212 of knowledgebase interfacer 204may receive an embedding vector representative of the user query from embedding vectorizer 210 and determine information in knowledgebase 108 that are semantically similar to the received embedding vector. As discussed above, sematic searcher 212 may determine similarities between the received embedding vector and a plurality of embedding vectors representative of the contents of knowledgebase 108 by calculating metrics such as, but not limited to, cosine distance and / or Euclidean distance. For instance given a user query, the embedding vector of the querv is compared to embedding vectors representative of the contents (e g., facts, concepts, relationships, rules, etc.) of knowledgebase 108, and the embedding vectors with the highest similarity scores (e.g., shortest, or smallest cosine and / or Euclidean distances) are considered the most semantically relevant. In embodiments, the plurality of indexed embedding vectors representative of the contents of knowledgebase 108 may be determined or calculated prior to receipt of the user query and stored at server(s) 104, knowledgebase 108, and / or any other location accessible to semantic searcher 212.

[0064] In step 406, information relevant to the first query are retrieved. For example, knowledgebase interfacer 204 may retrieve, from knowledgebase 108, information corresponding to the embedding vectors with the highest similarity scores to the embedding vector representative of the user query. In embodiments, knowledgebase interfacer 204 may provide the retrieved information to LLM interfacer 206.

[0065] Embodiments described herein may operate in various ways to convert a user query into a logic programming language query. For instance, FIG. 5 depicts a flowchart 500 of a method for employing an LLM to generate a logic programming language query, in accordance with an embodiment. Server(s) 104, LLM interfacer 206, prompt generator 216, and / or LLM querier 218 of FIGS. 1 and / or 2 may operate according to flowchart 500, for example. Note that not all steps of flowchart 500 may need to be performed in all embodiments, and in some embodiments, the steps of flowchart 500 may be performed in different orders than shown. Flowchart 500 is described as follows with respect to FIGS. 1 and 2 for illustrative purposes.

[0066] Flowchart 500 starts at step 502. In step 502, a prompt is generated to convert a first query into a logic programming language syntax based on information relevant to the first query. For example, prompt generator 216 may generate a prompt based on the user query received from query processor 202, knowledgebase interfacer 204, and / or componentsthereof, and information relevant to the first query received from knowledgebase interfacer 204, and / or components thereof. As discussed above, the generated prompt may request an LLM to convert the user query into a logic programming language syntax based on information (e.g., facts, concepts, relationships, rules, etc.) relevant to the user query. In embodiments, the information relevant to the user query may allow LLM 110 to infer a schema associated with knowledgebase 108.

[0067] In step 504, the prompt is provided to a large language model. For example, LLM querier 218 may provide the generated prompt to LLM 110. As discussed above, LLM 110 may be trained using a training dataset that includes information about programming languages (e.g., logic programming languages, object-oriented programming languages, etc.), and may be capable of generating logic programming language queries when prompted with information (e.g., facts, concepts, relationships, rules, etc.) about a knowledgebase to be queried.

[0068] In step 506, a logic programming language query is received from the large language model. For example, LLM querier 218 may receive, from LLM 110, a logic programming language query that is representative of the user query. In embodiments, LLM querier 218 may provide logic programming language query to knowledgebase querier 214 of knowledgebase interfacer 204.

[0069] Embodiments described herein may operate in various ways to automatically generate and / or populate a knowledgebase. For instance, FIG. 6 shows a block diagram of an example system 600 for automatic knowledgebase generation, in accordance with an embodiment. System 600 includes network(s) 106, and knowledgebase 108, as shown and described in conjunction with FIG. 1 and / or FIG. 2 above. As shown in FIG. 6, system 600 further includes one or more servers 602, and one or more data sources 604. In embodiments, server(s) 602 may reside in the same device and / or location as server(s) 104. In other embodiments, server(s) 602 may be separate and / or distinct from server(s) 104. As shown in FIG. 6, server(s) 602 may further include a knowledgebase generator 606 that includes an unstructured text analyzer 608, a relational database interfacer 610, and a knowledge graph analyzer 612. Each of the components of system 200 are described in detail as follows.

[0070] Server(s) 602 may include one or more of any server computing device suitable for performing functions that are ascribed thereto in the following description, as will beappreciated by persons skilled in the relevant art(s), including those mentioned elsewhere herein or otherwise known. Server(s) 602 may be implemented separately, or with reference to the exemplary computing environment 800 of FIG. 8 described below, server(s) 602 may alternatively be implemented as on-premises servers 892, and / or as part of network-based server infrastructure 870. In embodiments, server(s) 602 may include knowledgebase generator 606 that includes unstructured text analyzer 608, relational database interfacer 610, and knowledge graph analyzer 612. In embodiments, knowledgebase generator 606, unstructured text analyzer 608, relational database interfacer 610, and / or knowledge graph analyzer 612 may reside in the same device and / or location as UI manager 114, Q&A system 116, query processor 202, knowledgebase interfacer 204, LLM interfacer 206, response generator 208, embedding vectorizer 210, semantic searcher 212, knowledgebase querier 214, prompt generator 216, and / or LLM querier 218 of server(s) 104. In embodiments, knowledgebase generator 606, unstructured text analyzer 608, relational database interfacer 610, and / or knowledge graph analyzer 612 may reside in a separate device and / or location as UI manager 114, Q&A system 116, query processor 202, knowledgebase interfacer 204, LLM interfacer 206, response generator 208, embedding vectorizer 210, semantic searcher 212, knowledgebase querier 214, prompt generator 216, and / or LLM querier 218 of server(s) 104.

[0071] Data source(s) 604 may include, but are not limited to, one or more public or private databases or repositories that may comprise information, including, but not limited to, unstructured text (e.g., books, articles, periodicals, references, etc.), structured data (e.g., relational databases, webpages, knowledgebases, knowledge graphs, etc.), public information (e.g., published content, Internet sources, etc.), proprietary information (e.g., internal data, trade secrets, source code, product recipes or formulas, process information, sales numbers, employee data, production information, test results, protein sequences, market research, etc.), domain-specific information (e.g., information related to a particular topic or field).

[0072] Knowledgebase generator 606 may access data source(s) 604 to retrieve information contained therein, and generate data to be stored in knowledgebase 108. For instance, knowledgebase generator 606 may employ unstructured text analyzer 608, relational database interfacer 610, and / or knowledge graph analyzer 612 to analyze data from data source(s) 604 to determine facts, concepts, relationships, and / or rules containedtherein. In embodiments, data generated by knowledgebase generator 606 may be stored in knowledgebase 108 as logical statements or constructs (e.g., facts, concepts, relationships, rules, etc.). In embodiments, the logical statements, or constructs (e.g., facts, concepts, relationships, rules, etc.) may be stored in a relational database (e.g., SQL database, etc.).

[0073] Unstructured text analyzer 608 may analyze unstructured text from data source(s) 602 to determine facts, concepts, relationships, and / or rules contained therein. In embodiments, unstructured text analyzer 608 may apply natural language processing (NLP) techniques to unstructured textual data, which lacks a predefined structure or format in data, to extract meaningful information, patterns, and insights therefrom. NLP techniques include, but are not limited to, the use of text mining, infonnation retrieval, and sentiment analysis, to extract key entities (e.g., people, companies, organizations, drugs, objects, patents, grants, etc.), relationships (e.g., synonyms, ownership, assignments, licensing, rights, etc.), sentiments (e.g., reputation, reviews, etc.), and themes (e.g., domain, topic, market sector, etc.) from textual data. NLP analysis helps in discovering hidden patterns, gaining valuable insights, making data-driven decisions, and automating various tasks such as document classification, topic modeling, opinion mining, and summarization. Unstructured text analysis plays a crucial role in unlocking the value of unstructured data sources like social media posts, customer reviews, news articles, and other textual content, leading to enhanced understanding, knowledge extraction, and actionable intelligence. In embodiments, unstructured text analyzer 608 may generate tuples from the unstructured data. For example, unstructured text analyzer 608 may generate the following tuples from an article about a drug. drug ( 'aripiprazole ' ) dru ( 'Abilify' ) drug ( 'Aristada ' ) synonym ( 'aripiprazole ' , 'Abilify' ) synonym ( 'aripiprazole ' , 'Ari stada ' )

[0074] Relational database interfacer 610 may interact with relational databases associated with data source(s) 604 to extract data and relationships therefrom. In embodiments, relational database interfacer 610 may determine a schema of a relational database, andextract key entities (e.g., people, companies, organizations, drugs, objects, patents, grants, etc.), relationships (e.g., synonyms, ownership, assignments, licensing, rights, etc.), sentiments (e.g., reputation, reviews, etc.), and themes (e.g., domain, topic, market sector, etc.) from the data stored in the relational database.

[0075] Knowledge graph analyzer 612 may analyze knowledge graphs that organize information in a graph-like structure, consisting of nodes and edges. In embodiments, the nodes represent entities such as people, places, or concepts, while edges depict relationships between these entities. Knowledge graphs facilitate efficient querying, reasoning, and discovery of new insights by leveraging graph-based algorithms. In embodiments, knowledge graph analyzer 612 employs graph-based algorithms to extract key entities (e.g., people, companies, organizations, drugs, objects, patents, grants, etc.), relationships (e.g., synonyms, ownership, assignments, licensing, rights, etc.), sentiments (e.g., reputation, reviews, etc.), and themes (e.g., domain, topic, market sector, etc.) from the knowledge graphs.

[0076] Embodiments described herein may operate in various ways to automatically generate and / or populate a knowledgebase. For instance, FIGS. 7A-7C depict flowcharts 700A, 700B, and 700C, respectively, of methods for automatically generating a knowledgebase from various data sources, in accordance with an embodiment. Server(s) 602, knowledgebase generator 606, relational database interfacer 610, and / or knowledge graph analyzer 612 of FIG. 6 may operate according to flowchart(s) 700A, 700B, and / or 700C, for example.

[0077] Flowchart 700A starts at step 702. In step 702, relational tuples are automatically extracted from unstructured text. For example, unstructured text analyzer 608 may analyze unstructured textual data in data source(s) 604 to determine or generate tuples that represent relationships between key entities (e.g., people, companies, organizations, drugs, objects, patents, grants, etc.) described in the unstructured textual data.

[0078] Flowchart 700B starts at step 704. In step 704 data in a relational database are converted into logic programming statements. For example, relational database interfacer 610 may convert data stored in a relational database associated with data source(s) 604 into logic programming language statements. As discussed above, in embodiments, relational database interfacer 610 may determine a schema of the relational database, and extract key entities (e.g., people, companies, organizations, drugs, objects, patents, grants, etc.),relationships (e.g., synonyms, ownership, assignments, licensing, rights, etc.), sentiments (e.g., reputation, reviews, etc.), and themes (e.g., domain, topic, market sector, etc.) from the data stored in the relational database. Furthermore, relational database interfacer 610 may generate logic programming language statements based on information (e g., facts, concepts, relationships, rules, etc.) gleaned from the relational database.

[0079] Flowchart 700C starts at step 706. In step 706, a knowledge graph is converted into logic programming language statements. For example, knowledge graph analyzer 612 may analyze knowledge graphs in data source(s) 604 to determine entities and relationships organized in the knowledge graphs, and generate logic programming language statements representative of the entities and / or relationships. As discussed above, in embodiments, knowledge graphs may organize information in a graph-like structure, consisting of nodes representative of entities such as, but not limited to, people, places, or concepts, and edges representative of relationships between the entities. In embodiments, knowledge graph analyzer 612 may employ graph-based algorithms to extract key entities (e.g., people, companies, organizations, drugs, objects, patents, grants, etc.), relationships (e.g., synonyms, ownership, assignments, licensing, rights, etc.), sentiments (e.g., reputation, reviews, etc ), and themes (e g., domain, topic, market sector, etc ) from the knowledge graphs, and generate logic programming language statements based on information (e.g., facts, concepts, relationships, rules, etc.) gleaned from the knowledge graphs.III. Example Mobile Device and Computer System Implementation

[0080] The systems and methods described above in reference to FIGS. 1-7, including client(s) 102, server(s) 104, network(s) 106, knowledgebase 108, LLM 110, UI 112, UI manager 114, Q&A system 116, query processor 202, knowledgebase interfacer 204, LLM interfacer 206, response generator 208, embedding vectorizer 210, semantic searcher 212, knowledgebase querier 214, prompt generator 216, LLM querier 218, server(s) 602, data source(s) 604, knowledgebase generator 606, unstructured text analyzer 608, relational database interfacer 610, knowledge graph analyzer 612, and / or each of the components described therein, and / or the steps of flowcharts 300, 400, 500, 700A, 700B, and / or 700C may be implemented in hardware, or hardware combined with one or both of software and / or fimiware. For example, client(s) 102, server(s) 104, network(s) 106, knowledgebase108, LLM 110, UI 112, UI manager 114, Q&A system 116, query processor 202, knowledgebase interfacer 204, LLM interfacer 206, response generator 208, embedding vectorizer 210, semantic searcher 212, knowledgebase quener 214, prompt generator 216, LLM querier 218, server(s) 602, data source(s) 604, knowledgebase generator 606, unstructured text analyzer 608, relational database interfacer 610, knowledge graph analyzer 612, and / or each of the components described therein, and / or the steps of flowcharts 300, 400, 500, 700 A, 700B, and / or 700C may be each implemented as computer program code / instructions configured to be executed (e g., computer-executable) in one or more processors and stored in a computer readable storage medium. Alternatively, client(s) 102, server(s) 104, network(s) 106, knowledgebase 108, LLM 110, UI 112, UI manager 114, Q&A system 116, query processor 202, knowledgebase interfacer 204, LLM interfacer 206, response generator 208, embedding vectorizer 210, semantic searcher 212, knowledgebase querier 214, prompt generator 216, LLM querier 218, server(s) 602, data source(s) 604, knowledgebase generator 606, unstructured text analyzer 608, relational database interfacer 610, knowledge graph analyzer 612, and / or each of the components described therein, and / or the steps of flowcharts 300, 400, 500, 700A, 700B, and / or 700C may be implemented in one or more SoCs (system on chip). An SoC may include an integrated circuit chip that includes one or more of a processor (e.g., a central processing unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or further circuits, and may optionally execute received program code and / or include embedded firmware to cause a processor to perform functions.

[0081] Embodiments disclosed herein may be implemented in one or more computing devices that may be mobile (a mobile device) and / or stationary (a stationary' device) and may include any combination of the features of such mobile and stationary computing devices. Examples of computing devices in which embodiments may be implemented are described as follows with respect to FIG. 8. FIG. 8 shows a block diagram of an exemplary computing environment 800 that includes a computing device 802. In some embodiments, computing device 802 is communicatively coupled with devices (not shown in FIG. 8) external to computing environment 800 via network 804. Network 804 comprises one or more networks such as local area networks (LANs), wide area networks (WANs), enterprise networks, the Internet, etc., and may include one or more wired and / or wirelessportions. Network 804 may additionally or alternatively include a cellular network for cellular communications. Computing device 802 is described in detail as follows

[0082] Computing device 802 can be any of a variety of types of computing devices. For example, computing device 802 may be a mobile computing device such as a handheld computer (e.g., a personal digital assistant (PDA)), a laptop computer, a tablet computer (such as an Apple iPad™), a hybrid device, a notebook computer (e.g., a Google Chromebook™ by Google LLC), a netbook, a mobile phone (e.g., a cell phone, a smart phone such as an Apple® iPhone® by Apple Inc., a phone implementing the Google® Android™ operating system, etc.), a wearable computing device (e.g., a head-mounted augmented reality and / or virtual reality device including smart glasses such as Google® Glass™, Oculus Rift® of Facebook Technologies, LLC, etc.), or other type of mobile computing device. Computing device 802 may alternatively be a stationary computing device such as a desktop computer, a personal computer (PC), a stationary server device, a minicomputer, a mainframe, a supercomputer, etc.

[0083] As shown in FIG. 8, computing device 802 includes a variety of hardware and software components, including a processor 810, a storage 820, one or more input devices 830, one or more output devices 850, one or more wireless modems 860, one or more wired interfaces 880, a power supply 882, a location information (LI) receiver 884, and an accelerometer 886. Storage 820 includes memory 856, which includes non-removable memory 822 and removable memory 824, and a storage device 890. Storage 820 also stores an operating system 812, application programs 814, and application data 816. Wireless modem(s) 860 include a Wi-Fi modem 862, a Bluetooth modem 864, and a cellular modem 866. Output device(s) 850 includes a speaker 852 and a display 854. Input device(s) 830 includes a touch screen 832, a microphone 834, a camera 836, a physical keyboard 838, and a trackball 840. Not all components of computing device 802 shown in FIG. 8 are present in all embodiments, additional components not shown may be present, and any combination of the components may be present in a particular embodiment. These components of computing device 802 are described as follows.

[0084] A single processor 810 (e.g., central processing unit (CPU), microcontroller, a microprocessor, signal processor, ASIC (application specific integrated circuit), and / or other physical hardware processor circuit) or multiple processors 810 may be present in computing device 802 for perforating such tasks as program execution, signal coding, dataprocessing, input / output processing, power control, and / or other functions. Processor 810 may be a single-core or multi-core processor, and each processor core may be singlethreaded or multithreaded (to provide multiple threads of execution concurrently). Processor 810 is configured to execute program code stored in a computer readable medium, such as program code of operating system 812 and application programs 814 stored in storage 820. Operating system 812 controls the allocation and usage of the components of computing device 802 and provides support for one or more application programs 814 (also referred to as “applications” or “apps”). Application programs 814 may include common computing applications (e.g., e-mail applications, calendars, contact managers, web browsers, messaging applications), further computing applications (e.g., word processing applications, mapping applications, media player applications, productivity suite applications), one or more machine learning (ML) models, as well as applications related to the embodiments disclosed elsewhere herein.

[0085] Any component in computing device 802 can communicate with any other component according to function, although not all connections are shown for ease of illustration. For instance, as shown in FIG. 8, bus 806 is a multiple signal line communication medium (e.g., conductive traces in silicon, metal traces along a motherboard, wires, etc.) that may be present to communicatively couple processor 810 to various other components of computing device 802, although in other embodiments, an alternative bus, further buses, and / or one or more individual signal lines may be present to communicatively couple components. Bus 806 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures.

[0086] Storage 820 is physical storage that includes one or both of memory 856 and storage device 890, which store operating system 812, application programs 814, and application data 816 according to any distribution. Non-removable memory 822 includes one or more of RAM (random access memory), ROM (read only memory), flash memory, a solid-state drive (SSD), a hard disk drive (e.g., a disk drive for reading from and writing to a hard disk), and / or other physical memory device type. Non-removable memory 822 may include main memory and may be separate from or fabricated in a same integrated circuit as processor 810. As shown in FIG. 8, non-removable memory 822 stores firmware 818,which may be present to provide low-level control of hardware. Examples of firmware 818 include BIOS (Basic Input / Output System, such as on personal computers) and boot firmware (e.g., on smart phones). Removable memory 824 may be inserted into a receptacle of or otherwise coupled to computing device 802 and can be removed by a user from computing device 802. Removable memory 824 can include any suitable removable memory device type, including an SD (Secure Digital) card, a Subscriber Identity Module (SIM) card, which is well known in GSM (Global System for Mobile Communications) communication systems, and / or other removable physical memory device type. One or more of storage device 890 may be present that are internal and / or external to a housing of computing device 802 and may or may not be removable. Examples of storage device 890 include a hard disk drive, a SSD, a thumb drive (e.g., a USB (Universal Serial Bus) flash drive), or other physical storage device.

[0087] One or more programs may be stored in storage 820. Such programs include operating system 812, one or more application programs 814, and other program modules and program data. Examples of such application programs may include, for example, computer program logic (e.g., computer program code / instructions) for implementing one or more of client(s) 102, server(s) 104, network(s) 106, knowledgebase 108, LLM 110, UI 112, UI manager 114, Q&A system 116, query processor 202, knowledgebase interfacer 204, LLM interfacer 206, response generator 208, embedding vectorizer 210, semantic searcher 212, knowledgebase querier 214, prompt generator 216, LLM querier 218, server(s) 602, data source(s) 604, knowledgebase generator 606, unstructured text analyzer 608, relational database interfacer 610, knowledge graph analyzer 612, along with any components and / or subcomponents thereof, as well as the flowcharts / flow diagrams (e.g., flowcharts 300, 400, 500, 700A, 700B, and / or 700C) described herein, including portions thereof, and / or further examples described herein.

[0088] Storage 820 also stores data used and / or generated by operating system 812 and application programs 814 as application data 816. Examples of application data 816 include web pages, text, images, tables, sound files, video data, and other data, which may also be sent to and / or received from one or more network servers or other devices via one or more wired or wireless networks. Storage 820 can be used to store further data including a subscriber identifier, such as an International Mobile Subscriber Identity (IMSI), and anequipment identifier, such as an International Mobile Equipment Identifier (IMEI). Such identifiers can be transmitted to a network server to identify users and equipment.

[0089] A user may enter commands and information into computing device 802 through one or more input devices 830 and may receive information from computing device 802 through one or more output devices 850. Input device(s) 830 may include one or more of touch screen 832, microphone 834, camera 836, physical keyboard 838 and / or trackball 840 and output device(s) 850 may include one or more of speaker 852 and display 854. Each of input device(s) 830 and output device(s) 850 may be integral to computing device 802 (e.g., built into a housing of computing device 802) or external to computing device 802 (e.g., communicatively coupled wired or wirelessly to computing device 802 via wired interface(s) 880 and / or wireless modem(s) 860). Further input devices 830 (not shown) can include aNatural User Interface (NUI), a pointing device (computer mouse), a joystick, a video game controller, a scanner, a touch pad, a stylus pen, a voice recognition system to receive voice input, a gesture recognition system to receive gesture input, or the like. Other possible output devices (not shown) can include piezoelectric or other haptic output devices. Some devices can serve more than one input / output function. For instance, display 854 may display information, as well as operating as touch screen 832 by receiving user commands and / or other information (e.g., by touch, finger gestures, virtual keyboard, etc.) as a user interface. Any number of each type of input device(s) 830 and output device(s) 850 may be present, including multiple microphones 834, multiple cameras 836, multiple speakers 852, and / or multiple displays 854.

[0090] One or more wireless modems 860 can be coupled to antenna(s) (not shown) of computing device 802 and can support two-way communications between processor 810 and devices external to computing device 802 through network 804, as would be understood to persons skilled in the relevant art(s). Wireless modem 860 is shown generically and can include a cellular modem 866 for communicating with one or more cellular networks, such as a GSM network for data and voice communications within a single cellular network, between cellular networks, or between the mobile device and a public switched telephone network (PSTN). Wireless modem 860 may also or alternatively include other radio-based modem types, such as a Bluetooth modem 864 (also referred to as a “Bluetooth device”) and / or Wi-Fi modem 862 (also referred to as an “wireless adaptor”). Wi-Fi modem 862 is configured to communicate with an access point or otherremote Wi-Fi-capable device according to one or more of the wireless network protocols based on the IEEE (Institute of Electrical and Electronics Engineers) 802.11 family of standards, commonly used for local area networking of devices and Internet access. Bluetooth modem 864 is configured to communicate with another Bluetooth-capable device according to the Bluetooth short-range wireless technology standard(s) such as IEEE 802.15.1 and / or managed by the Bluetooth Special Interest Group (SIG).

[0091] Computing device 802 can further include power supply 882, LI receiver 884, accelerometer 886, and / or one or more wired interfaces 880. Example wired interfaces 880 include a USB port, IEEE 1394 (FireWire) port, a RS-232 port, an HDMI (High-Definition Multimedia Interface) port (e.g., for connection to an external display), a DisplayPort port (e.g., for connection to an external display), an audio port, an Ethernet port, and / or an Apple® Lightning® port, the purposes and functions of each of which are well known to persons skilled in the relevant art(s). Wired interface(s) 880 of computing device 802 provide for wired connections between computing device 802 and network 804, or between computing device 802 and one or more devices / peripherals when such devices / peripherals are external to computing device 802 (e.g., a pointing device, display 854, speaker 852, camera 836, physical keyboard 838, etc ). Power supply 882 is configured to supply power to each of the components of computing device 802 and may receive power from a battery internal to computing device 802, and / or from a power cord plugged into a power port of computing device 802 (e.g., a USB port, an A / C power port). LI receiver 884 may be used for location determination of computing device 802 and may include a satellite navigation receiver such as a Global Positioning System (GPS) receiver or may include other type of location determiner configured to determine location of computing device 802 based on received information (e.g., using cell tower triangulation, etc.). Accelerometer 886 may be present to determine an orientation of computing device 802.

[0092] Note that the illustrated components of computing device 802 are not required or all-inclusive, and fewer or greater numbers of components may be present as would be recognized by one skilled in the art. For example, computing device 802 may also include one or more of a gyroscope, barometer, proximity sensor, ambient light sensor, digital compass, etc. Processor 810 and memory 856 may be co-located in a same semiconductor device package, such as being included together in an integrated circuit chip, FPGA, or system-on-chip (SOC), optionally along with further components of computing device 802.

[0093] In embodiments, computing device 802 is configured to implement any of the above-described features of flowcharts herein. Computer program logic for performing any of the operations, steps, and / or functions descnbed herein may be stored in storage 820 and executed by processor 810.

[0094] In embodiments, computing device 802 is configured to implement a feedback mechanism to learn from previous responses to improve future responses. The feedback may include, but is not limited to, internal feedback, external feedback, a combination of internal and external feedback, and / or the like without any limitation to the described embodiments.

[0095] In some embodiments, server infrastructure 870 may be present in computing environment 800 and may be communicatively coupled with computing device 802 via network 804. Server infrastructure 870, when present, may be a network-accessible server set (e.g., a cloud-based environment or platform). As shown in FIG. 8, server infrastructure 870 includes clusters 872. Each of clusters 872 may comprise a group of one or more compute nodes and / or a group of one or more storage nodes. For example, as shown in FIG. 8, cluster 872 includes nodes 874. Each of nodes 874 are accessible via network 804 (e g., in a “cloud-based” embodiment) to build, deploy, and manage applications and services. Any of nodes 874 may be a storage node that comprises a plurality of physical storage disks, SSDs, and / or other physical storage devices that are accessible via network 804 and are configured to store data associated with the applications and services managed by nodes 874. For example, as shown in FIG. 8, nodes 874 may store application data 878.

[0096] Each of nodes 874 may, as a compute node, comprise one or more server computers, server systems, and / or computing devices. For instance, a node 874 may include one or more of the components of computing device 802 disclosed herein. Each of nodes 874 may be configured to execute one or more software applications (or “applications”) and / or services and / or manage hardware resources (e.g., processors, memory, etc.), which may be utilized by users (e.g., customers) of the network-accessible server set. For example, as shown in FIG. 8, nodes 874 may operate application programs 876. In an implementation, a node of nodes 874 may operate or comprise one or more virtual machines, with each virtual machine emulating a system architecture (e g., an operating system), in an isolated manner, upon which applications such as application programs 876 may be executed.

[0097] In an embodiment, one or more of clusters 872 may be co-located (e.g., housed in one or more nearby buildings with associated components such as backup power supplies, redundant data communications, environmental controls, etc.) to form a datacenter, or may be arranged in other manners. Accordingly, in an embodiment, one or more of clusters 872 may be a datacenter in a distributed collection of datacenters. In embodiments, exemplary computing environment 800 comprises part of a cloud-based platform such as Amazon Web Services® of Amazon Web Services, Inc. or Google Cloud Platform™ of Google LLC, although these are only examples and are not intended to be limiting.

[0098] In an embodiment, computing device 802 may access application programs 876 for execution in any manner, such as by a client application and / or a browser at computing device 802. Example browsers include Microsoft Edge® by Microsoft Corp, of Redmond, Washington, Mozilla Firefox®, by Mozilla Corp, of Mountain View, California, Safari®, by Apple Inc. of Cupertino, California, and Google® Chrome by Google LLC of Mountain View, California.

[0099] For purposes of network (e.g., cloud) backup and data security, computing device 802 may additionally and / or alternatively synchronize copies of application programs 814 and / or application data 816 to be stored at network-based server infrastructure 870 as application programs 876 and / or application data 878. For instance, operating system 812 and / or application programs 814 may include a file hosting service client, such as Microsoft® OneDrive® by Microsoft Corporation, Amazon Simple Storage Service (Amazon S3)® by Amazon Web Services, Inc., Dropbox® by Dropbox, Inc., Google Drive™ by Google LLC, etc., configured to synchronize applications and / or data stored in storage 820 at network-based server infrastructure 870.

[0100] In some embodiments, on-premises servers 892 may be present in computing environment 800 and may be communicatively coupled with computing device 802 via network 804. On-premises servers 892, when present, are hosted within an organization’s infrastructure and, in many cases, physically onsite of a facility of that organization. Onpremises servers 892 are controlled, administered, and maintained by IT (Information Technology) personnel of the organization or an IT partner to the organization. Application data 898 may be shared by on-premises servers 892 between computing devices of the organization, including computing device 802 (when part of an organization) through a local network of the organization, and / or through further networks accessible to theorganization (including the Internet). Furthermore, on-premises servers 892 may serve applications such as application programs 896 to the computing devices of the organization, including computing device 802. Accordingly, on-premises servers 892 may include storage 894 (which includes one or more physical storage devices such as storage disks and / or SSDs) for storage of application programs 896 and application data 898 and may include one or more processors for execution of application programs 896. Still further, computing device 802 may be configured to synchronize copies of application programs 814 and / or application data 816 for backup storage at on-premises servers 892 as application programs 896 and / or application data 898.

[0101] Embodiments described herein may be implemented in one or more of computing device 802, network-based server infrastructure 870, and on-premises servers 892. For example, in some embodiments, computing device 802 may be used to implement systems, clients, or devices, or components / subcomponents thereof, disclosed elsewhere herein. In other embodiments, a combination of computing device 802, network-based server infrastructure 870, and / or on-premises servers 892 may be used to implement the systems, clients, or devices, or components / subcomponents thereof, disclosed elsewhere herein.

[0102] As used herein, the terms “computer program medium,” “computer-readable medium,” and “computer-readable storage medium,” etc., are used to refer to physical hardware media. Examples of such physical hardware media include any hard disk, optical disk, SSD, other physical hardware media such as RAMs, ROMs, flash memory, digital video disks, zip disks, MEMs (microelectronic machine) memory, nanotechnology-based storage devices, and further types of physical / tangible hardware storage media of storage 820. Such computer-readable media and / or storage media are distinguished from and nonoverlapping with communication media and propagating signals (do not include communication media and propagating signals). Communication media embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wireless media such as acoustic, RF, infrared and other wireless media, as well as wired media. Embodiments are also directed to such communication media thatare separate and non-overlapping with embodiments directed to computer-readable storage media.

[0103] As noted above, computer programs and modules (including application programs 814) may be stored in storage 820. Such computer programs may also be received via wired interface(s) 880 and / or wireless modem(s) 860 over network 804. Such computer programs, when executed or loaded by an application, enable computing device 802 to implement features of embodiments discussed herein. Accordingly, such computer programs represent controllers of the computing device 802.

[0104] Embodiments are also directed to computer program products comprising computer code or instructions stored on any computer-readable medium or computer-readable storage medium. Such computer program products include the physical storage of storage 820 as well as further physical storage types.IV. Conclusion

[0105] While various embodiments have been described above, it should be understood that they have been presented by way of example only, and not limitation. It will be understood by those skilled in the relevant art(s) that various changes in form and details may be made therein without departing from the spirit and scope of the described embodiments as defined in the appended claims. Accordingly, the breadth and scope of the present embodiments should not be limited by any of the above-described exemplary embodiments, but should be defined in accordance with the following claims and their equivalents.

Claims

AMENDED CLAIMS received by the International Bureau on 06 November 2024 (06.11.2024)WHAT IS CLAIMED IS:1 A method for generating logic programming language queries, comprising: receiving a first query; determining a first embedding vector representative of the semantic meaning of the first query; performing a semantic search on the knowledgebase using a first embedding vector to determine information relevant to the first query; generating, using a large language model, a logic programming language query based at least on the first query and the information relevant to the first query; executing the logic programming language query against a knowledgebase to generate results; and returning the results as a response to the first query.2 (Cancelled).3 The method of claim 1, wherein said determining information relevant to the first query comprises generating the information relevant to the first query using a machine learning model.4 The method of any of claims 1 or 3, wherein said generating, using a large language model, a logic programming language query comprises: generating a prompt to convert the first query into a logic programming language syntax based on the information relevant to the first query; providing the prompt to the large language model; and receiving the logic programming language query from the large language model.5 The method of any of claims 1 or 3-4, wherein the logic programming language query is at least one of: a Prolog query; a Datalog query; a Cypher query; or39AMENDED SHEET (ARTICLE 19)a Loglisp query.

6. The method of any of claims 1 or 3-5, wherein the information relevant to the first query comprises at least one of: a predicate; a fact; or a rule.7 The method of any of claims 1 or 3-6, wherein the knowledgebase is generated based on at least one of: automatic extraction of relational tuples from unstructured text; conversion of data in a relational database into logic programming language statements; or conversion of a knowledge graph into logic programming language statements.8 A system for generating logic programming language queries, comprising: a processor; a memory device stores program code structured to cause the processor to: receive a first query; determine a first embedding vector representative of the semantic meaning of the first query; perform a semantic search on the knowledgebase using a first embedding vector to determine information relevant to the first query; generate, using a large language model, a logic programming language query based at least on the first query and the information relevant to the first query; execute the logic programming language query against a knowledgebase to generate results; and return the results as a response to the first query.9 (Cancelled).40AMENDED SHEET (ARTICLE 19)10. The system of claim 8, wherein, to determine information relevant to the first query, the program code is further structured to cause the processor to generate the information relevant to the first query using a machine learning model.

11. The system of any of claims 8 or 10, wherein, to generate the logic programming language query, the program code is further structured to cause the processor to: generate a prompt to convert the first query into a logic programming language syntax based on the information relevant to the first query; provide the prompt to the large language model; and receive the logic programming language query from the large language model.

12. The system of any of claims 8 or 10-11, wherein the logic programming language query is at least one of: a Prolog query; a Datalog query; a Cypher query; or a Loglisp query.

13. The system of any of claims 8 or 10-12, wherein the information relevant to the first query comprises at least one of: a predicate; a fact; or a rule.

14. The system of any of claims 8 or 10-13, wherein the knowledgebase is generated based on at least one of: automatic extraction of relational tuples from unstructured text; conversion of data in a relational database into logic programming language statements; or conversion of a knowledge graph into logic programming language statements.

15. A computer-readable storage medium comprising computer-executable instructions, that when executed by a processor, cause the processor to:41AMENDED SHEET (ARTICLE 19)receive a first query; determine a first embedding vector representative of the semantic meaning of the first query; perform a semantic search on the knowledgebase using a first embedding vector to determine information relevant to the first query; generate, using a large language model, a logic programming language query based at least on the first query and the information relevant to the first query; execute the logic programming language query against a knowledgebase to generate results; and return the results as a response to the first query.

16. (Cancelled).

17. The computer- readable storage medium of claim 15, wherein, to determine information relevant to the first query, the instructions, when executed by the processor, further cause the processor to generate the information relevant to the first query using a machine learning model.

18. The computer- readable storage medium of any of claims 15 or 17, wherein, to generate the logic programming language query, the instructions, when executed by the processor, further cause the processor to: generate a prompt to convert the first query into a logic programming language syntax based on the information relevant to the first query; provide the prompt to the large language model; and receive the logic programming language query from the large language model.

19. The computer- readable storage medium of any of claims 15 or 17-18, wherein the logic programming language query is at least one of: a Prolog query; a Datalog query; a Cypher query; or a Loglisp query.42AMENDED SHEET (ARTICLE 19)20. The computer- readable storage medium of any of claims 15 or 17-19, wherein the information relevant to the first query comprises at least one of: a predicate; a fact; or a rule.43AMENDED SHEET (ARTICLE 19)

Images