System prompt generation for LLM to convert NLQ to SQL

Abstract

A computer-implemented method is provided of generating a system prompt for a large language model to convert a natural language query (NLQ) to a structured query language(SQL). 5 The method includes generating (1606) the system prompt for the NLQ based on information extracted from a first database. The information identified from a knowledge graph includes a structured view of data including telecommunications domain knowledge and relations within the data filtered by a chain of thought prompt that represents a logical relation between a plurality of nodes in the knowledge graph. The method further includes passing (1614) the system prompt to 0 the LLM to convert the NLQ to the SQL.

Description

System Prompt Generation for LLM to Convert NLQ to SQLTECHNICAL FIELD

[0001] The present disclosure relates generally to computer-implemented methods performed by a computing device for generating a system prompt for a large language model (LLM) to convert a natural language query (NLQ) to a structured query language (SQL), and related methods and devices.BACKGROUND

[0002] Telecommunication network operations generate and maintain huge datasets in multiple layers of a telecommunication stack. To analyze operational issues, bugs, or a network incident, for example, data needs to be collected from different sources.

[0003] Within the telecommunications industry, expanding fifth generation (5G) networks may bring exponentially growing data volumes and complexity that may need to be captured and analyzed in an efficient way.SUMMARY

[0004] There currently exist certain challenges. In some domains, such as a telecommunications network, a database (DB) may not be static, and may contain vast amounts of data including at least some dynamically changing data. Analysis and / or extraction of data from the DB may be needed to analyze and / or manage operational issues or network incidents, for example. Exponentially growing amounts of data and data complexity, however, may make it difficult to capture or analyze data in an efficient manner. While some approaches may try to use a LLM, use of an LLM in some domains, such as the telecommunications industry for example, may include challenges. Challenges may include, for example, challenges with preservation of data integrity and protection against security breaches; system prompt formulation that takes into account domain-specific nuances; incomprehensible or difficult system prompts; intensive and complex use of resources (e.g., computing resources); insufficient accuracy rates of an LLM (e.g., due to hallucinations), particularly for specific domains; and / or training an LLM with appropriate domain knowledge.

[0005] Certain aspects of the disclosure and their embodiments may provide solutions to these or other challenges.

[0006] Some embodiments provide a computer-implemented method of generating a system prompt for a large language model to convert a NLQ to a SQL. The method includes generating the system prompt for the NLQ based on information extracted from a first database. The information identified from a knowledge graph includes a structured view of data including telecommunications domain knowledge and relations within the data filtered by a chain of thought prompt that represents a logical relation between a plurality of nodes in the knowledge graph. The method further includes passing the system prompt to the LLM to convert the NLQ to the SQL.

[0007] Other embodiments provide a computing device. The computing device comprises at least one processor; and at least one memory connected to the at least one processor and storing program code that is executed by the at least one processor to perform operations. The operations include to generate the system prompt for the NLQ based on information extracted from a first database. The information identified from a knowledge graph includes a structured view of data including telecommunications domain knowledge and relations within the data filtered by a chain of thought prompt that represents a logical relation between a plurality of nodes in the knowledge graph. The operations further include to pass the system prompt to the LLM to convert the NLQ to the SQL.

[0008] Some embodiments include a computer program product comprising a non-transitory storage medium including program code to be executed by at least one processor of a computing device configured to generate a system prompt for a LLM to convert a NLQ to a SQL. Execution of the program code causes the computing device to perform operations. The operations comprise to generate the system prompt for the NLQ based on information extracted from a first database. The information identified from a knowledge graph includes a structured view of data including telecommunications domain knowledge and relations within the data filtered by a chain of thought prompt that represents a logical relation between a plurality of nodes in the knowledge graph. The operations further include to pass the system prompt to the LLM to convert the NLQ to the SQL.

[0009] Based on generating a system prompt for a NLQ based on information extracted from a first DB, where the information is identified from a knowledge graph including a structured view of data including telecommunications domain knowledge and relations within the data filtered by a chain of thought prompt that represents a logical relation between a plurality of nodes in the knowledge graph, certain embodiments may provide one or more of the following technical advantages. Accuracy because the LLM can create an accurate SQL based on the NLQ; data security because the data resides outside the LLM; providing appropriate domain knowledge; efficient use of data (e.g., token buffer size) for system prompt generation; accuracy (e.g., lackof / decreased hallucinations) based on an efficient amount of data fed to generate the system prompt; and dynamically generating a comprehensible SQL that corresponds to the NLQ.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this application, illustrate certain non-limiting embodiments of the present disclosure. In the drawings:

[0011] Figure 1 is a block diagram illustrating an example method of system prompt generation based on a dynamically generated tailored knowledge graph according to some embodiments;

[0012] Figure 2 is a block diagram illustrating another example of a method and system according to some embodiments;

[0013] Figure 3 is schematic diagram of an example of an entity relationship diagram for a first DB schema according to some embodiments;

[0014] Figure 4 is a schematic drawing of an entity relationship diagram for a second DB schema according to some embodiments;

[0015] Figure 5 is a schematic drawing of an entity relationship diagram for a third DB schema according to some embodiments;

[0016] Figure 6 is a block diagram showing an example of extraction of knowledge a relationship between nodes according to some embodiments;

[0017] Figure 7 is a flow chart of operations for preparing an example knowledge graph according to some embodiments;

[0018] Figures 8A and 8B are schematic drawings of generating a knowledge graph of a DB schema according to some embodiments;

[0019] Figure 9 is aflow chart of operations for an example using aNLQ-SLQ pair to generate few-shot examples according to some embodiments;

[0020] Figure 10 is a flow chart of operations of an example for masking / encoding sensitive information according to some embodiments;

[0021] Figure 11 is a flow chart of operations of an example for generating a dynamic system prompt according to some embodiments;

[0022] Figure 12 is a flow chart of operations of an example for dynamically generating a system prompt according to some embodiments;

[0023] Figure 13 is a sequence diagram of an example of generating a knowledge graph according to some embodiments;

[0024] Figure 14 is a sequence diagram of an example of dynamically generating an NLQ- SLQ pair for few-shot example generation with a change of DB schema or for a new DB schema according to some embodiments;

[0025] Figure 15 is a sequence diagram of an example of dynamically generating a system prompt according to some embodiments;

[0026] Figures 16 and 17 are flow charts illustrating operations of a computing device according to some embodiments;

[0027] Figure 18 is a block diagram of a communication system in accordance with some embodiments; and

[0028] Figure 19 is a block diagram of computing device according to some embodiments.DETAILED DESCRIPTION

[0029] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art, in which examples of embodiments of the present disclosure are shown. Inventive concepts may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present / used in another embodiment.

[0030] A telecommunication network includes two types of data: static data where the frequency of updates to this type of data can be very low, and dynamic data which is changing in nature. The data can be structured or unstructured, but the analyzed data may be stored in a relational database (DB) where the analyzed data can be rendered by using a business intelligence (BI) tool or another form to a user to get insight into the network. For example, a subscriber- oriented network analytics product designed for cloud-native fourth generation (4G) and fifth generation (5G) packet core networks may be used; and / or an analytics tool may provide communication service providers (CSPs) with tools to try to manage huge amounts of data and try to efficiently target issues at a subscriber level as they occur.

[0031] With the evolution of artificial intelligence (Al) LLMs, a LLM may be used in various aspects of analytics and telecommunication operations.

[0032] The application of LLMs in developing Al products may be increasing. One of the areas of application of a LLM is to dynamically convert a NLQ to a SQL. The converted SQL can be executed with the help of backend database tables and generates an answer to the NLQ.

[0033] In the context of NLQ to SQL transformation, various LLMs models may be developed with massive amounts of data points. Using transformer-based architectures, for example, each LLM may also incorporate many model parameters and significant computing capacity may be needed to execute and fine tune the LLM for the development of a customer application. Some approaches conclude that to get the right SQL query for a given NLQ, an appropriate prompt(s) needs to be passed to the LLM. Different LLMs, however, accept prompts in a different style and format. The smart use of a LLM, therefore, may be in the effectiveness of interactions with the LLM based on intelligent prompt engineering. For certain NLQs that require a complex SQL query with multiple joins, nested queries, and group by statements to execute, getting an appropriate answer from the NLQ can depend on how dynamically the prompt template is designed and passed to the LLM.

[0034] An analytics database may be huge and include a plurality of tables, and each table can include a plurality of columns / fields. Thus, it may be difficult to identify and dynamically generate a correct DB schema (e.g., table and columns) based on an input NLQ. An inaccurately understood DB schema is not helpful to generate an appropriate and optimized SQL query, for example.

[0035] While manually storing explicit information for each underlying table description, the corresponding column descriptions, and various domain-specific information helps the dynamic prompt generation functions to generate an appropriate and precise DB schema. Along with this, dynamically generated few shot examples (also referred to herein as shot prompts) drive the LLM to generate the right SQL query, such an approach may not be possible. For example in the telecommunications industry, rapid evolution means that the tables, and the associated columns are not static. Rather, they change as per business requirements and technology evolution, for example. Thus, manual augmentation of the underlying tables and corresponding column descriptions may not be a feasible approach to use the full capability of NLQ to SQL transformation via LLM-based AL An adaptive approach, however, may be lacking.

[0036] Moreover, it may be very difficult to get a trained open-source LLM with telecommunications knowledge, especially because so much information may be proprietary.

[0037] Additionally, DB design can be more art than science. As a consequence, it can be very difficult to obtain needed domain knowledge information from a DB schema because additional information may be needed, for example.

[0038] Integrating the use of an LLM in the telecommunications industry may include challenges, such as (1) providing an LLM with appropriate domain knowledge; (2) token buffer sizes for system prompt engineering; (3) an LLM having a propensity towards hallucinations versus an appropriate amount of data to feed during system prompt engineering; (4) preservation of data integrity and protection against security breaches. For example, the susceptibility to prompt injection, whereby malicious actors exploit vulnerabilities to manipulate model outputs, coupled with the potential loss of proprietary design insights, may underscore a need for safeguarding organizational knowledge, data, and / or intellectual property (“sensitive data” leakage to the LLM); and (5) changes to a DB, such as tables in the DB, for a telecommunications environment.

[0039] A further challenge may include equipping the LLM with knowledge that takes into account the above challenges through system prompt engineering.

[0040] Amidst such an intricate landscape, the exploration of generative Al (Gen Al) may be of interest in the pursuit of optimizing a 5G network experience. However, this may necessitate a detailed examination of the feasibility of harnessing Gen Al, specifically LLMs, for the conversion of a NLQ into a SQL which may facilitate substantially easier access to analyzed data housed within a telecommunications analytics platforms, for example.

[0041] In some approaches, an LLM includes data and the data can include proprietary domain specific knowledge and / or product knowledge. A method is lacking for keeping an LLM separate from domain specific knowledge and / or product knowledge. Such a method, however, may be desirable due to the evolving nature of LLMs and security challenges and costs involved in training an LLM.

[0042] A method may be desirable that secures organizational product knowledge and uses an LLM in a smart way to build next generation products, for example.

[0043] However, the nascent nature of LLM technology may present challenges. Challenges may include, for example, that achieving proficiency in discerning domain-specific nuances within a telecommunications lexicon may require a concerted effort in prompt formulation, informed by deep domain expertise. Training an LLM withNLQs, accompanied by corresponding DB schemas and SQL queries, may be fraught with resource-intensive complexities and cost.

[0044] Moreover, the application of an LLM within this context may represent a novel frontier within the Gen Al landscape including, for example, being characterized by limitations such as attaining high accuracy rates, particularly when tailored for specific domains. Addressing such a challenge may include imbuing an LLM with contextual awareness and domain-specific vocabulary, intricately aligned with an underlying DB schema.

[0045] The versatility of end-users in posing NLQs in diverse formats may further complicate matters, which may underscore a need for sophisticated prompting processes. Processes may be lacking, however, that can effectively translate proprietary domain knowledge into comprehensible prompts for an LLM to enable nuanced comprehension and proficient SQL generation. For example, the use of an LLM in SQL generation may be lacking and there may be challenges in understanding less explored domains where product specifications and domain specific knowledge are less available.

[0046] In some approaches, user prompts are not based on mathematics but rather on human intention, for example, and this may be the way LLM is trained. System prompt engineering includes a process to generate a result from the LLM. However, generating a system prompt through a defined method is lacking. Moreover, the explainability of an LLM itself may be a new domain and the conversion of a NLQ to a SQL may not have matured enough to use it in production.

[0047] Further challenges in use of LLMs in real-time application development may include that, when working with a pre-trained LLM, the LLM is not necessarily trained with domain specific terms and knowledge. Additionally or alternatively, use by a general user not trained to interact with the LLM can be challenging.

[0048] Moreover, inference may be a challenge when trying to use a LLM more purposefully. Challenges can include one or more of the following.

[0049] Ambiguity and Contextual Understanding: LLMs can struggle with understanding the context and nuances of a user's NLQ. The LLM may misinterpret ambiguous questions or require additional context to provide an accurate response.

[0050] Correctness and Reliability: LLMs may generate responses that are factually incorrect or lack reliability. Ensuring that the information provided by the LLM is accurate can be a challenge.

[0051] Biased Responses: LLMs can inadvertently produce biased or inappropriate responses. Addressing bias and trying to ensure that the LLM's outputs align with ethical standards may be important.

[0052] Response Length: LLMs can generate responses that are excessively long or verbose, making it challenging for users to extract relevant information efficiently.

[0053] Relevance: LLMs may generate responses that contain irrelevant information, requiring post-processing to extract the pertinent details.

[0054] Consistency: Maintaining consistency in responses, especially in multi -turn conversations for example, can be challenging, as LLMs may provide contradictory answers.

[0055] Language and Terminology: LLMs may not use the desired language or terminology specific to a domain. Ensuring that the LLM aligns with the language conventions of the application's domain can be vital.

[0056] Resource Intensity: Performing inference with LLMs can be resource-intensive, requiring substantial computational power and infrastructure.

[0057] Certain aspects of the disclosure and their embodiments may provide solutions to these or other challenges. Some examples generate a dynamic tailored knowledge graph based on design documents, specifications, and DB schema. Some examples include the convergence of advanced system prompting methodologies with knowledge graph-based information extraction. Such examples may include Gen Al and telecommunications convergence and, thus, may leverage cutting-edge technologies to address complex telecommunications industry challenges.

[0058] Some examples include dynamically generating the NLQ and its corresponding SQL, where data is extracted from DB design documents, entity diagrams, normalization information, and / or other requirements relating to the design of the DB. The extracted data can include DB table descriptions, column descriptions, and / or relevant domain-specific information. The extracted data can be parsed during intelligent prompt engineering activity so that the LLM can create a SQL query based on the NLQ.

[0059] Some examples include a trained or untrained LLM generating complex SQL queries via focused prompting of the LLM, including the use of dynamic knowledge graph information to accompany the NLQ. As a consequence accuracy of the LLM in generating the SQL may be improved.

[0060] In some examples, ontologies are created based on provided DB design inputs as well as a DB schema. A dynamically created knowledge graph may then be used to prepare focused statements of domain knowledge and may also create example SQL queries. In addition to providing focused prompting, such examples may address the issue of proprietary or sensitive data by separating domain-specific knowledge from the LLM, thus allowing a trained or untrained LLM to prepare the SQL query responses.

[0061] Generation of table and column / field descriptions is included in some examples. The table and column / field descriptions can be used in successful system prompt engineering to generate a system prompt dialect that can cater to varying degrees of complex SQL query generation from a NLQ.

[0062] Generation of domain specific statement generation is included in some examples. For example, based on a user NLQ, a statement(s) specific to a telecommunications domain may bedynamically generated. The system prompt dialect may be robust for handling complex SQL queries.

[0063] A NLQ / SLQ pair is generated in some examples. A NLQ can be dynamically generated based on provided input documents and a DB schema. The corresponding SQL can be generated using a LLM service. The generated NLQ and SQL pair can be used to select a few-shot example(s) in the context of dynamic system prompt generation for facilitating in-context learning. Due to the nature of a changing DB schema (e.g., business and table schemas), a fixed set of NLQ and SQL pairs may not be sufficient to apply as a knowledge base in the examples for choosing appropriate few-shot examples. Thus, some example include updated the NLQ / SQL pair from time to time.

[0064] Dynamic chain of thought (CoT) prompt generation for complex relations is included in some examples. CoT prompting may improve the performance of the LLM by prompting the LLM, with a CoT prompt(s), to explain its reasoning before arriving at an answer. Relations are included between tables and entities in a DB schema. Some examples include dynamically generating a sequence of relational statements based on which CoT prompts can be generated to understand the relations between the entities. These relational statements can be the part of the system prompt dialect to generate an appropriate SQL query from the NLQ. The knowledge graph edge descriptions which represent the connection(s) between two nodes can be used to represent the relation of the nodes in a statement(s). If there are connecting edges between multiple nodes consecutively, the relations of those nodes can be represented by the statements. These statements are integral parts of representing the CoT prompt for the LLM.

[0065] For example, a logical statement of relation based on a connecting edge of nodes in the knowledge graph are used to create a CoT. The CoT can be used to select, e.g., precisely select, a DB schema from the DB schema that is previously selected with the help of a keyword and intention (e.g., meaning and / or context) of the keyword. In an example, the NLQ includes a keyword “cell” which is present in multiple tables of the DB. From the initial selected DB schema all schemas are selected, but now the intention or context of the NLQ and key are used and checked / filtered with the CoT prompt to determine whether all of the multiple tables containing the keyword “cell” really are related as per the CoT prompt. Based on these logical statement(s) of relation, a precise schema(s) is selected. The CoT, which is a representation of a complex DB schema for example, can be included as a part of the system prompt if the selected DB schema refers to, e.g., multiple tables and the relations between the columns of those tables are complex in nature. The inclusion of the CoT can be based on the complexity of the NLQ and the selected DB schema (e.g., multiple tables and columns). An example is as follows.

[0066] In this example, a DB schema includes two tables. A first table (KPI table) holds raw data for key performance indicators (KPIs), capturing data for respective cells every 15 minutes. The second table (e.g., a threshold table) includes threshold values for each KPI and cell, organized by hour and day of the week. A goal is to compare the actual KPI values with the thresholds for each cell and generate alarms if any conditions are breached.

[0067] To do this, in this example, both tables are joined using a primary key from the KPI table (e.g., node cell id) with the node cell id from the Threshold table (which acts as a foreign key).

[0068] In this example, a customer wants to see the trend of actual KPI values versus threshold values for each cell over the last week.

[0069] To accomplish this, details from both tables are provided, including their primary and foreign keys along with additional information such as the hourly thresholds unique to each cell and KPI, to LLM as a part of a dynamic system prompt. This information is extracted from a knowledge graph generated using database design documents, low-level design (LLD), and entityrelationship (ER) diagrams. By providing this information in this example, a good quality response is obtained from the LLM without hallucination.

[0070] In some examples, a system prompt is generated through inclusion of a knowledge graph framework that serves as a foundational pillar for information synthesis and extraction. Such an approach may not only enrich the system prompt dialect but also may empower a LLM with a nuanced understanding of domain-specific knowledge, thereby augmenting the efficacy and accuracy of SQL generation.

[0071] Certain embodiments may provide one or more of the following technical advantages. In contrast to approaches for LLM training that require that relevant knowledge and data resides inside the LLM and, thus, brings with it risks of exposing legally protected and sensitive data to public cloud LLMs, examples herein may improve data security based on data being external to the LLM. In some examples, domain and product specific knowledge are kept secure in a secure premises and an LLM is used to help to extract the answer from the secure storage. Additionally, in some examples, the system prompt can be dynamically generated along with masking to secure the information and to provide optimal information to increase the accuracy of the LLM to prepare the SQL.

[0072] Moreover, based on generating the system prompt dialect in a smart way using information extracted from a knowledge graph, relevant / optional information may be provided to the LLM to generate a response to the NLQ having improved accuracy.

[0073] Figure 1 is a block diagram illustrating an example method of system prompt 116 generation 104 based on a dynamically generated 102 tailored knowledge graph 100. As shown in Figure 1 by the dotted line representing an information separator 114 (e.g., which also may include masking as discussed further herein), the LLM 118 is separate from data 106, 108, 110 that includes domain knowledge, customer specific knowledge, and / or vendor specific product knowledge. As a consequence, the data 106, 108, 110 is kept secure and apart from the LLM 118. This differs from other approaches where an LLM generates an answer from its own training data and, thus, may compromise data security.

[0074] In an example using the method of Figure 1, actual customer specific information or secure data resides in a different DB that can be, for example, on the customer’s premises. Thus, data security risks may be reduced when a general-purpose public cloud LLM 118 is used.

[0075] As discussed further herein, additional masking and aliasing tools can be used to further enhance data security.

[0076] LLM 118, thus, is used to generate a correct, accurate, or acceptable SQL 120 to extract the data from the remote, secure DB, residing at a customer premises for example, whereas LLM 118 can reside in a public cloud. Thus, a general purpose LLM 118 can be used to answer a domain or customer specific NLQ 112 whereas the LLM 118 is not trained with that data. The additional masking / alias can be used to generate more security between the actual knowledge and the knowledge shared to LLM 118 to extract the actual answer for the user.

[0077] Figure 2 is a block diagram illustrating another example of a method and system according to some embodiments. A user (e.g., via a user equipment (UE)) makes aNLQ. A system prompt for the NLQ is generated based on the dynamically generated knowledge graph 100. The system prompt is passed to the LLM to convert the NLQ to a SQL. As shown, the LLM is separate from a proprietary DB, text embeddings, and / or a vector DB that includes domain knowledge, customer specific knowledge, and / or vendor specific product knowledge. As a consequence, the proprietary DB, text embeddings, and / or vector DB is kept secure and apart from the LLM.

[0078] In the example in Figure 2, actual customer specific information or secure data resides in the proprietary DB, text embeddings, and / or the vector DB. Thus, data security risks may be reduced when a foundation LLM is used.

[0079] In the example in Figure 2, the LLM is used to generate a correct, accurate, or acceptable SQL to extract data, via the data framework for the LLM, from the remote proprietary DB, text embeddings, and / or vector DB. Thus, the foundation LLM can be used to provide a response to a domain or customer specific NLQ from the user, whereas the LLM is not trained with that data.

[0080] The dynamic tailored knowledge graph 100 includes data organization and analysis, and is characterized by its adaptability and specificity. Unlike traditional, static knowledge graphs, the dynamic version of examples herein can evolve and reconfigure itself in response to new information, user interactions, and / or changes in its environment.

[0081] The tailored aspect of the knowledge graph refers to the knowledge graph being specifically structured to cater to distinct domains, user requirements, and / or application-specific needs, for example.

[0082] Such a configuration may make the knowledge graph exceptionally effective for targeted data analysis, personalized information retrieval, and / or intelligent decision-making processes, for example. A dynamic tailored knowledge graph can be an intelligent data processing tool that may offer precision and adaptability in handling complex, ever-changing data landscapes such as a telecommunications domain.

[0083] A computer-implemented method of some examples includes training to prepare domain knowledge based optimal statements to use in system prompt engineering where the LLM will be used to interact with a DB, such as a relational DB.

[0084] The computer-implemented method of some examples includes obtaining a training data set that includes text data. The text data indicates one or more phrases or sentences of database design inputs. The DB design inputs include statements and description of elimination data redundancy from DB requirements, identify a defined objective of the DB, and / or sample data objects and information stored in the DB.

[0085] Input to the method of such examples can include an entity diagram, information of primary key and foreign keys of each entity, and designing of a relation which can help to identify the relationships among entities.

[0086] Types of relationship include, without limitation: one-to-many; many-to-many; and / or one-to-one.

[0087] The input to the method of these examples also can include a refinement and normalization of the design, for example one or more of where any additional columns were used, create a new table for optional data using a one-to-one relationship, split a large table into two smaller tables, etc.

[0088] Normalization can include the first normal form (INF), second normal form (2NF), third normal form (3NF), higher normal forms and corresponding entities (e.g., tables) and a description of the cause to realize the normalization.

[0089] Integrity rules of the entity (e.g., tables) also may be included in the input. Integrity rules can include one or more of an entity integrity rule and a referential integrity rule.

[0090] An example entity integrity rule is that the primary key cannot contain NULL. Otherwise, the primary key cannot uniquely identify the row; and / or for a composite key made up of several columns, none of the column can contain NULL. A relational database management system (RDBMS) can check and enforce this rule.

[0091] An example referential integrity rule is that each foreign key value must be matched to a primary key value in the table referenced (or parent table). A row with a foreign key can be inserted in the child table only if the value exists in the parent table. If the value of the key changes in the parent table (e.g., the row updated or deleted), all rows with this foreign key in the child table(s) must be handled accordingly, (a) The changes can be disallowed; (b) the change can be cascaded (or the records deleted) in the child tables accordingly; or (c) the key value in the child tables can be set to NULL.

[0092] A RDBMS can be setup to perform the check and ensure the referential integrity in the specified manner. In addition to the above two general integrity rules, a business logic integrity rule can be included. For example, there can be integrity, such as validation, pertaining to the business logic, e.g., zip code shall be 5-digit within a certain range; delivery date and time shall fall in the business hours; quantity ordered shall be equal or less than quantity in stock, etc. A business logic integrity rule can be carried out in validation rule, such as for a specific column, or programming logic.

[0093] Input of example methods also can include the description of the integrity, such as why certain integrity is required for that data set. Indexing information of the entities also can be included in the input.

[0094] The method of some examples can use a transformer type neural network architecture that helps computers understand natural language including entity diagrams and the inputs described herein. The method of some examples also can include training a classifier using supervised machine learning based on the training data set, the finalized DB schema design and additional text data in the form of statements, such as domain knowledge statements,) from the inputs.

[0095] Example methods can include preparing statements from input DB design documents and a final DB schema discussed further herein. The statements can be used for system prompt engineering to convert the NLQ to a SQL.

[0096] Example methods can further include providing a training methodology along with the template to generate an automated system prompt. The system prompt can be used for LLM inference to create simple and complex SQL.

[0097] Some examples include an apparatus including at least one processor; and memory coupled to the at least one processor and storing instructions that, when executed by the processor, cause the at least one processor to perform operations including to obtain a training data set including text data indicating one or more phrases or sentences the entity diagram, requirement documents and final schema of the DB and the generated knowledge or prompt example from the training set. The operations can further include to train a classifier using supervised machine learning based on the training data set and additional text data indicating one or more domain knowledge-based prompt related to DB schema which can be used as additional supportive prompt to generate a more accurate query.

[0098] Some example method include generating a knowledge graph based on the provided DB schema and the design inputs of the DB schema. The knowledge graph can be used to create a sample NLQ in a different variation and a corresponding SQL query. These SQL queries can be a standard SQL query which can serve the purpose of fetching data from that particular DB schema, keeping the knowledge graph as a reference.

[0099] The ontologies generated NLQ and corresponding SQL reference and the generated knowledge statement can be used in the system prompt preparation along with the user input prompt of the LLM.

[0100] As discussed herein, there are practical challenges to using a LLM in a real-world product or solution, including for example a telecommunications product or solution.

[0101] An LLM can convert text to SQL. However, the training cost of the LLM is high and if training is desired with telecommunications domain specific knowledge along with vendor specific knowledge, training can be even more challenging in terms of computing cost and time.

[0102] Token size also can be a challenge when using an open source LLM model. If a high token size LLM is used, a hardware requirement may be large which can increase the cost. On the other hand if more token are passed, the unnecessary token, e.g. information, may lead to a hallucinated result.

[0103] In some examples, in converting text to SQL, the LLM is used to generate the SQL to extract the knowledge from external DB where the actual answer is present in the external DB. The knowledge is not present in the LLM itself. Thus, accuracy of the generated SQL in terms of a telecommunications domain or product specific DB is important.

[0104] As discussed herein, the LLM can create the SQL the text along with DB schema and few shot examples are passed. However, in real life the DB schema can be huge. A system can have 100s of tables and each table also can have 100s of columns, for example. Out of these 100sof tables and columns, many column names can be similar or can be repetitive across many tables. As a consequence, this can lead to the LLM producing hallucinated results.

[0105] Some approaches to passing minimum information include a retrieval augmented generation (RAG) based approach. However, it may be challenging to prepare the RAG knowledge based in terms of any specific domain or product, for example.

[0106] In contrast, an example method includes generating the knowledge graph for any domain or product, such as a telecommunication domain or product. The example method generates a knowledge graph that can be used in system prompt engineering to get a more accurate prompt dialect to communicate with the LLM.

[0107] The following are examples of challenges and aspects of methods herein .

[0108] In a first example, the NLQ is “What are the 4G cells available in X region?”. In logically thinking about the SQL query, the SQL query should contain a condition statement. The condition statement should contain a where clause which should map the 4G to a particular column and the X region should also map to another column. While the LLM may be capable to create a syntactical correct SQL, in the product context, the semantic correctness is more important. For example, which column the LLM should consider for 4G and X region.

[0109] A design document of the DB can have this information. For example, “the Network can have multiple supported radio access technology (RAT) type (e.g., 3G, 4G, 5G, wifi)”. If this knowledge can be used to create a system prompt statement, the LLM can have the attention to process the information.

[0110] An example of components of the system prompt dialect is as follows:<DB schema><Few-shot example><CoT statements to represent the relations of the schema(s)><4G is a RATtype><What are the 4G cells available in X region?>

[0111] In this example, a similar way that the X region can be mapped to any column in any table also is which has to be considered. The column name can be “Area” to denote the region in the DB schema. The region may be present in a different relational table and the column name may be “Area”. In that case, the relation between tables also may be important to create the attention which will help LLM to create the SQL. The method of this example can help to create the attention, and along with it, the method can provide a CoT that represents the table relation.This CoT prompt can help to create a complex SQL for a complex DB-schema, which may be common in a telecommunication domain’s product.

[0112] A second example considers a question “What are average throughput in X2 interface?”. Human behavior and construction of questions typically is different for different humans. For this question, for example, consider that “X2” interface is present in the DB. Thus, based on the knowledge graph, the X2 can be mapped with a particular column to create the conditional statement. A different human, however, can state the same question is a different way, such as “What are average throughput between two eNodeB?” While the “X2” interface is basically the interface between two eNodeB, this information is not available to the LLM.

[0113] If the domain knowledge and requirement documents used as an input to create the DB schema have this sort information, the LLM can create the attention and create the correct SQL. Example methods herein help to create a knowledge graph representing the relation between tables and columns, along with domain-specific knowledge, automatically for each column and thought behind the creating DB schema for that product. This can help to create correct conditions and a semantically correct SQL to extract the information from the product DB.

[0114] In a third example, an evaluation of LLMs to create a syntactical and semantically correct SQL for a hardcore telecommunication domain specific DB schema found that there were multiple tables with almost the same DB schema, but the tables contained different data sets. For example, a table may have a subset of data where a table had some KPI columns representing the total value for a day, and another similar table included the same KPI but contained the data for only 5 minutes. For a NLQ to extract the data for one day, the LLM may chose the 1-day table or it may aggregate the 5 minute table to create the response.

[0115] Another example of multiple tables with almost the same DB schema, but the tables contain different data sets includes multiple KPI tables for different radio access technology (RAT) types. For example, a KPI name may be the same for third generation (3G), fourth generation (4G) / 5G. but the DB schema may have a different table for each RAT. The column names in each of these different tables may be similar or mostly similar. However, in a real scenario when a user asks a question there is a purpose behind the question. For example, the user may want to know an answer to a question that happens to be specific to a particular column or a particular table. In this example, the user is asking for a KPI for a particular RAT.

[0116] According to examples herein, a knowledge graph is prepared for automated system prompt engineering. According to examples, the LLM will understand the context and attention of the question, and the corresponding DB and column, based on which the LLM generates the SQL query and condition clause in the SQL. To do so, in examples herein, design documents andrequirement documents are used as input, along with other domain knowledge, to generate the knowledge graph. The knowledge graph is used to generate a dynamic system prompt based on NLQ.

[0117] Thus, some examples herein include two steps: (1) generating a dynamic tailored knowledge graph, statements, and CoTs; and (2) generating a dynamic system prompt based on use of the knowledge graph, statements, and CoTs.

[0118] Generating the dynamic tailored knowledge graph, statements, and CoTs uses a DB design document as an input. The DB design document can include the entity relationship diagram of the proposed DB schema including, for example, a relationship between tables and columns of tables. In some examples, the entity relationship diagram for the proposed DB schema is a diagram that is generated and shows the relation between tables and how a column(s) is connected with another column(s) of a different table, based on a foreign key (fk), for example. Figure 3 is schematic diagram of an example of an entity relationship diagram for a first DB schema.

[0119] Some examples include generating or modifying the entity relationship and including a description of the relation based on domain knowledge. In some examples, additional information is extracted that is behind the finalized DB schema based on a requirement, DB design, and / or domain knowledge. The additional information can be included in the entity diagram as a description of the relation, as shown in the example in Figure 4 of an entity relationship diagram for a second DB schema.

[0120] Some examples include describing a relation in terms of, for example, “how, why, what, and / or when” and including the description of the knowledge, as shown in the example in Figure 5 of an entity relationship diagram for a third DB schema.

[0121] Thus, some examples include generating knowledge of the entity relation of the DB schema. The knowledge can include the cause of a relation extracted from a DB design, for example. The entity relationship diagram can include a description of each column and why and how, for example, the column is related to a different column(s) or different table(s).

[0122] The knowledge can include a description of respective columns used in the DB schema that is generated, along with the context of the respective columns, according to some examples. This knowledge can be used to generate the dynamic system prompt. The following is an example of generated descriptions of columns according to some examples:

[0123] Some examples further include generating a statement of entity relations. In a complex relational DB, for example, multiple tables can be connected by well-designed foreign keys. The statement of relation of the entities can be generated along with an example and cause behind the relation. For example, domain knowledge, e.g. an entity can have multiple values. In that case, there may be two tables created, the initial table as the key and another table having the values as rows mapped to initial table. This statement can be used as a CoT or additional system prompting to create an accurate SQL query based on the NLQ, domain knowledge and the used DB schema.

[0124] In some examples, the input can further include a DB requirement and design document(s). For example, if a DB is designed to take inputs or have dependencies on any domain specific information, then those specifications are retrieved as an input to an Al- model used toextract the information of relations of the DB schema based on an attention mechanism. An example is as follows.

[0125] Figure 6 is a block diagram showing an example of extraction of knowledge a relationship between nodes. To extract the knowledge from textual data 600, it may be essential to enable a computing device to comprehend natural language. This can involve employing one or more natural language processing (NLP) techniques, such as sentence segmentation 602, dependency parsing, parts-of-speech tagging 604, and / or entity recognition 606. These techniques can play an important role in breaking down and understanding the structure and meaning of text. Sentence segmentation 602 can help in dividing text into meaningful units, while dependency parsing can reveal the relationships between words. Parts-of-speech tagging 604 can assign grammatical categories to words, which can aid in understanding their roles within sentences. Entity recognition or extraction 606 can identify and classify specific entities mentioned in the text, such as names of people, organizations, locations, etc. Together, these NLP techniques can lay the foundation for building a comprehensive and insightful knowledge graph from textual information. Initially, in some example, respective nodes (e.g., columns) can be used to create attention in the provided documents and an encoder decoder model can be used to build the description of the respective nodes (e.g., column / table).

[0126] As shown in Figure 6, the generated knowledge of the respective nodes (e.g., column / table) can be used to build 612 the relation between any two nodes and to understand the relationship between them denoted by an edge of the knowledge graph. In some examples, using an encoder-decoder architecture of a model and provided description of connected nodes, the model can convert the input to a word embedding and then, using an attention mechanism 608, the model can extract the relationship 610, generate 612 the relationship between different columns and associate the relationship with the edge of two nodes as shown in the example in Figure 6.

[0127] An LLM can be used to extract the knowledge by using instruction-based prompt engineering, for example. The LLM can utilize an attention mechanism 608 to identify the relationship between different nodes, such as columns and tables, as shown in the following example prompt template:Relationship between respective columns

[0128] Some examples use the DB schema and prepare a knowledge graph of columns and tables defined as nodes. The edge of each connection can be used to hold information about the connectivity. The extracted knowledge can be used to define the relation between two nodes in the knowledge graph. In some examples, the model can extract the information based on how, why, when and / or what type of questions of probable relations between two nodes in the knowledge graph.

[0129] Further, in some examples, additional information of tables is extracted. For example, the additional information can include refinement and normalization of the DB design indicating where any additional columns were used, creating a new table for optional data using a one-to-one relationship, split a large table into two smaller tables, etc. and keeping the additional information associated with the tables (e.g., with nodes in the knowledge graph).

[0130] The nodes representing columns also can hold the extracted description of the cause to realize the normalization of respective columns and tables and keep them associated with the columns (e.g., nodes).

[0131] Figure 7 is a flow chart of operations for preparing an example knowledge graph according to some examples.

[0132] In operation 1, a DB requirement, DB design document, and / or domain specifications are input to an Al model.

[0133] In operation 2, based on these inputs, the Al model generates a graph columns associated with tables as shown in the example in Figure 8A.

[0134] In operation 3, the Al model prepares the possible connections for the columns present in the graph as shown in the example in Figure 8A.

[0135] The Al model, in operation 4, extracts knowledge as a statement related to respective columns based on use of the DB design, DB requirement, and domain specification and keeps it associated with the columns. The domain specification can include, for example, a specification of a telecommunications domain and a specification for a telecommunications product.

[0136] In operation 5, the Al model finds logical relations which may be how, why, what, and / or when statements of respective logical connections of columns in the relationship diagram, and keeps them in the connectors as shown in the example in Figure 8B.

[0137] The Al model, in operation 6, reads the respective connections based on a primary key and foreign key defined in the DB schema and prepares logical statements. In operation 7, the Al model defines a statement of connection logical statement(s) and specific knowledge of respectivecolumns and keeps the statement(s) with columns in the knowledge graph as shown in the example in Figure 8B

[0138] In operation 8, the type of relationship of respective relations (that is, connections) of columns is extracted. The relations may be one-to-many, many-to-many, one-to-one, for example. The information is kept in the connection associated with the connector as shown in the example in Figure 8B.

[0139] In operation 9, additional information of the tables is extracted, including refinement and normalization of the DB design, for example. The additional information can include, for example, where any additional columns were used, creation of a new table for optional data using a one-to-one relationship, a split of a large table into two smaller tables, etc. The additional information is kept associated with the tables as shown in the example in Figure 8B.

[0140] In operation 10, the description of the cause is stored to realize normalization of respective columns and tables, and the description is kept associated with the columns as shown in the example in Figure 8B.

[0141] As shown in the example in Figure 8B, the final knowledge graph provides a structured view of the data and its relations; and stores the logical relationships between the columns and tables.

[0142] Figure 9 is a flow chart of operations for an example using aNLQ-SLQ pair to generate few-shot examples (also referred to herein as a few-shot prompts). Shot prompting (which also may be referred to as example prompting) refers to constructing a prompt(s) that includes an example(s) of input-output pairs for the LLM. The example(s) can guide the LLM by illustrating how to approach and respond to a particular question or task. Thus, a LLM may work better with a shot example(s). However, it may be difficult to dynamically create a knowledge repository of few-shot examples and use the knowledge repository to create a dynamic system prompt. As shown in Figure 9, in some examples, the knowledge graph is used to generate a sample NLQ that is a variation of the NLQ by using the descriptions of the columns, and a relation(s) of the respective columns, as a reference. As shown in block 900, a machine learning (ML) model along with attention can be used to generate a set of sample NLQs for respective columns. The ML model can take a column name and its description from the knowledge graph and, from that description, create the sample NLQs. The sample NLQs are associated with the columns. The list of sample NLQs and respective corresponding answers are stored in corresponding columns associated with the sample NLQs. This process can be repeated for various columns based on the knowledge graph.

[0143] As shown in block 902, the LLM passes, to a NLQ-to-SQL generator, the respective sample NLQs and corresponding columns and column descriptions to generate a respective SQL corresponding to the respective sample NLQs. Since there are no assumptions and the sample NLQ, DB schema and conditions are known, the corresponding SQL is correct. In other words, because the sample NLQs are related to the given columns, the generated SQL uses the correct DB schema. Thus, the SQL is correct and can be used to extract correct information. The sample NLQ and corresponding SQL are stored, for example, in a list of NLQ-SQL pairs.

[0144] As shown in block 904, the operations of blocks 900 and 902 are repeated to create additional sample NLQs from the column descriptions and connector descriptions, and the additional sample NLQs along with the columns are passed to the LLM to create the corresponding SQLs in a recursive way for the possible combinations in the knowledge graph.

[0145] A purpose of generating a dynamic system prompt include to keep product or other sensitive information / knowledge away from the LLM. As a result, any LLM can be used, for example an LLM in an organization / company, for the purpose of knowledge extraction analysis and generating reports.

[0146] Thus, based on the inclusion of the dynamically generated system prompt, the security of knowledge may be ensured even though a separate LLM can be used to get insights from the data. Moreover, the LLM does not need to be to train with the domain knowledge or any product specific knowledge. The knowledge, therefore, is secured and the LLM is used to prepare the SQL to extract the knowledge which can reside locally.

[0147] Some examples include identifying product specific or other sensitive information from the knowledge graph; and masking the sensitive information. For example, sensitive information in the names and descriptions of columns name may be masked or encoded.

[0148] Figure 10 is a flow chart of operations of an example for masking / encoding sensitive information. In block 1000, the knowledge graph and the system prompt are inputs; and, in block 1002, the system prompt is checked for sensitive information, such as specific product information. If sensitive information is identified, the sensitive information is marked. In block 1004, the marked sensitive information is masked or replaced by an alias before the system prompt is sent, e.g., to a system prompt interface.

[0149] Masking or encoding the sensitive information can help to generate a secure system prompt that can be used for inference in the LLM and is secured, while not sharing the sensitive information to generate the SQL.

[0150] A graph-based query / search can be used to extract information stored in the knowledge graph to generate the system prompt. The extracted knowledge from the knowledge graph can beused directly to build the dynamic system prompt, or the extracted knowledge can be from a search of the knowledge graph and stored in vector DB to identify semantic similarity of sample NLQs along with few-shot examples. In some examples, generating the dynamic system prompt is based on use of the knowledge graph, the statements, and CoTs.

[0151] Figure 11 is a flow chart of operations of an example for generating a dynamic system prompt. In block 1100, a vector DB is created based on the generated column descriptions along with the columns. In block 1102, the connector descriptions with associated columns are extracted / s earched from the vector DB. In block 1104, NLQ-SQL pairs are extracted / searched from the vector DB. In block 1106, statements of domain knowledge and CoT logical relations between columns and tables are extracted / searched from the vector DB.

[0152] Figure 12 is a flow chart of operations of an example for dynamically generating a system prompt. The system prompt is generated based on the received 1200 NLQ. In block 1202, the NLQ is used to extract a keyword and an intention of the keyword from the NLQ. In block 1204, a DB schema is found by using the keyword and the intention of the keyword from the knowledge graph, which may provide closely matched columns. In block 1206, a cosine similarity score is calculated between each keyword and few-shot examples created by the NLQ-SQL pairs. Based on the complexity of the statements, in block 1208, additional few shot examples may be included from a list prepared by a domain expert, such as additional knowledge discussed herein. In block 1210, LLM and CoT prompting is used to create a schema for the input NLQ. The dynamic DB schema is created using the LLM and CoTs prompting by proving the NLQ and fewshots examples.

[0153] In block 1212, based on the finalized DB schema, domain knowledge statements, a logical relation between the columns, and final few shot examples are fetched.

[0154] In block 1214, the DB schema, few-shot examples, knowledge statements, and logical relations between columns are used along with the NLQ are used as a system prompt to generate the SQL.

[0155] The generated SQL is used to extract the knowledge and prepare the answer to the NLQ. The data storage, e.g., DB, can reside in its own premises.

[0156] Figure 13 is a sequence diagram of an example of generating a knowledge graph. As shown, Figure 13 includes a document importer 1300, a knowledge graph generator 1302, and an LLM 1304.

[0157] The document importer 1300, in this example, is a component that takes a DB design documents, specification document and / or DB requirement documents as input. Document importer 1300 processes the input documents as required and identifies tables and a DB schema.

[0158] Knowledge graph generator 1302, in this example, is a component that represents the DB schema in a graph and creates edges between the nodes. The knowledge graph generator 1302 may be able to add additional nodes and description, and modify the graph with respect to the change of input. Knowledge graph generator 1302 also can share the specific extracted information for each node with the LLM 1304, and the output can be updated in nodes or edges of the graph based on type (e.g., description of the node or description of the relation of the nodes).

[0159] LLM 1302, in this example, uses a transformer encoder-decoder architecture that leverages both encoder and decoder layers to process and generate text. The encoder can capture the context of the input text, and the decoder can generate the output text based on the encoded context. This architecture may excel in tasks requiring context understanding and sequence generation and identifying the relations based on the context.

[0160] In operation 1, based on document importer 1300 providing to knowledge graph generator 1302 inputs including one or more of a DB requirement, DB design document, and / or domain specifications, knowledge graph generator 1302 identifies entity diagrams and generates a graph with tables and columns as parent-child nodes, as shown for example in Figure 8 A.

[0161] In operation 2, the knowledge graph generator 1302 prepares the possible connections for the nodes comprising columns present in the graph.

[0162] In operation 3, document importer 1300 provides to LLM 1304 knowledge related to respective nodes from the DB design document, requirement document, and / or specification. Each node can be shared as part of a prompt.

[0163] In operation 4, LLM 1304 extracts a description of the nodes from the knowledge.

[0164] In operation 5, LLM 1304 provides the extracted knowledge to knowledge graph generator 1302; and knowledge graph generator 1302 updates the knowledge graph nodes with knowledge and adds the description.

[0165] In operation 6, document importer 1300 provides to LLM 1304 the possible relations between each node based on, for example, how, why, what, and / or when, etc. statements.

[0166] In operation 7, LLM 1304 finds the how, why, what, and / or when, etc. statements of respective logical connections of nodes; and knowledge graph generator 1302 updates the knowledge graph with the relations between nodes.

[0167] Figure 14 is a sequence diagram of an example of dynamically generating an NLQ- SLQ pair for few-shot example generation with a change of DB schema or for a new DB schema. As shown, Figure 14 includes a graph traverser 1400, LLM 1304, and an NLQ-SQL engine 1402.

[0168] Graph traverser 1400, in this example, is a component that can traverse the graph that shares the description of respective nodes along with a prompt to generate sample NLQs forrespective nodes. Generally, graph traverser 1400 will share all of the possible combinations of nodes and their respective descriptions with LLM 1304 to generate the possible sample NLQs. Graph traverser 1400 also can share the relations of nodes as a part of the prompt so that the sample NLQs corresponding to that can also be generated by using the LLM 1304.

[0169] NLQ-SQL engine 1402, in this example, is a component for NLQ to SQL generation. NLQ-SQL engine 1402 can prepare the final dynamic system prompt as well. NLQ-SQL engine 1402 can interact with a RAG and vector DB to prepare the final system prompt dialect. NLQ- SQL engine 1402 also can store the NLQ-SQL pair, which can be generated by passing the generated sample NLQ and corresponding nodes of the graph, that is the DB schema, as a part of prompt to prepare the NLQ-SQL pair for dynamic few-shot example generation.

[0170] In operation 1, graph traverser 1400 provides to LLM 1304 nodes of the knowledge graph along with attached respective descriptions of the nodes; and the LLM 1304 prepares possible sample NLQs. As noted in Figure 14, LLM 1304 receives possible nodes and combinations of those nodes.

[0171] In operation 2, LLM 1304 provides to NLQ-SQL engine 1402, the sample NLQs along with identity of the input table(s) and column(s) as nodes.

[0172] A loop is performed that includes operations 3 and 4. In operation 3, NLQ-SQL engine 1402 provides to LLM 1304 the identities of the table(s) and column(s), that is nodes, of the knowledge graph and the corresponding sample NLQ to prepare a SQL. The sample NLQ can be a SQL generation prompt.

[0173] In operation 4, LLM 1304 provides to NLQ-SQL engine 1402 the sample NLQs and corresponding SQLs along with the input table(s) and node(s), that is the auto-generated NLQ- SQL pair for few-shot examples.

[0174] Figure 15 is a sequence diagram of an example of dynamically generating a system prompt. As shown, Figure 15 includes NLQ-SQL engine 1402 and LLM 1304. NLQ-SQL engine 1402, in this example, generates the final system prompt, verifies the system prompt, and validates a token size.

[0175] In operation 1, NLQ-SQL engine 1402 provides the input NLQ to LLM 1304.

[0176] In operation 2, LLM 1304 extracts a keyword(s) and an intend on(s) of the keyword(s) from the NLQ.

[0177] In operation 3, LLM 1304 provides the keyword(s) and intention(s) of the NLQ to NLQ-SQL engine 1402.

[0178] A loop of operations 4 and 5 is performed. In operation 4, NLQ-SQL engine 1402 finds a DB schema using the keyword(s) and the intention(s) of the keyword(s) based on the knowledge graph; and provides the DB schemata LLM 1304.

[0179] In operation 5, LLM 1304 calculates a similarity score, e.g. a cosine similarity score, between a keyword and finds a few-shot example(s) created by a NLQ-SQL pair. Based on the complexity of the NLQ, the NLQ-SQL pair(s) can be used as source for selecting additional fewshot examples. In operation 6, NLQ-SQL engine 1402 adds any additional few-shot examples to handle a special or specific case based on the complexity and intention of the NLQ. Special or specific cases may be exceptional cases, for example.

[0180] In operation 7, NLQ-SQL engine 1402 identifies a DB schema based on CoT. The LLM 1304 generates the final DB schema that will be part of the dynamic system prompt.

[0181] In operation 8, LLM 1304 provides the DB schema to NLQ-SQL engine 1402.

[0182] In operation 9, NLQ-SQL engine 1402, based on the final DB schema, few-shot examples and intention(s) of the NLQ, finds domain specific statements from the vector DB. The vector DB can be a proprietary vector DB.

[0183] In operation 10, based on the complexity of the schema, the logical statements of relation in the final DB schema, NLQ-SQL engine 1402 fetches extracted knowledge and node relationships, as discussed with respect to Figure 6 for example.

[0184] In operation 11, NLQ-SQL engine 1402 generates a system prompt. The system prompt also is verified and token size validation is performed.

[0185] In operation 12, NLQ-SQL engine 1402 passes the system prompt to LLM 1304..

[0186] In operation 13, LLM 1304 converts the NLQ to a SQL.

[0187] While some approaches includes a retrieval augmented generation (RAG) DB that serves as a static DB that may aid a LLM in extracting relevant contextual information, such an approach lack dynamic updating. In contrast, examples herein include dynamic updating of a DB, e.g., a RAG DB, in response to changes in DB design. Moreover, examples herein may enhance a RAG knowledge DB with evolving domain-specific proprietary knowledge, which may be important for successful implementation of a NLQ to SQL query manager but often may be overlooked.

[0188] In contrast to approaches using a static RAG DB, examples herein involves creating a knowledge graph that is used in prompt engineering. This may help in generating more accurate system prompt dialects for effective communication with a hosted LLM.

[0189] Further, in some approaches, relationships may exist between tables and entities in existing DB schema. Examples herein further dynamically generate a sequence of relationalstatements. These statements may facilitate the creation of CoT prompts to decipher inter-entity relationships. These relational statements become part of the prompt dialect and, thus, may enable the accurate translation of NLQs into correct SQL queries.

[0190] LLMs may be well suited to work in any hypervisor environment for a high computing resource requirement. The methodology of examples herein may be implementable in a public and / or a private cloud infrastructure.

[0191] Figure 16 is a flow chart illustrating operations of a computing device according to some embodiments. As shown, some embodiments are directed to a computer-implemented method of generating a system prompt for a LLM to convert a NLQ to a SQL. In block 1606, the method includes generating the system prompt for the NLQ based on information extracted from a first database. The information is identified from a knowledge graph including a structured view of data including telecommunications domain knowledge and relations within the data filtered by a CoT prompt that represents a logical relation between a plurality of nodes in the knowledge graph. As shown in block 1614, the method further includes passing the system prompt to the LLM to convert the NLQ to the SQL.

[0192] A logical relation between tables and columns in the first database can be used to prepared CoT prompt in the case of complex DB schema, for example. The CoT prompt may be used as a part of the system prompt if the NLQ refers to that DB schema and the intention (e.g., meaning / context) of the NLQ is same as mentioned in the logical relations and corresponding DB schema.

[0193] The plurality of nodes may include a plurality of tables respectively including a column. The knowledge graph may include the plurality of nodes and a connecting edge between at least two nodes from the plurality of nodes. The connecting edge can represent at least one logical relation between the plurality of nodes.

[0194] In some embodiments, the LLM may use the at least one logical relation to prepare at least one statement of logical relation and may use the at least one statement of logical relation to prepare the CoT prompt; and, optionally, the method may further include receiving (1604) the CoT prompt.

[0195] The CoT may include a plurality of logical statements of relations that represents a complex selected database schema. The complex selected database schema may include a plurality of tables and complex relations between columns of the plurality of tables. The CoT may be included in the system prompt based on the complexity of the NLQ and the complex selected database schema.

[0196] In some embodiments, the first database includes one or more schemas respectively including one or more tables including a column.

[0197] In some embodiments, generating the system prompt in block 1606 includes (1) receiving a keyword and a meaning or context of the keyword extracted from the NLQ; (2) finding a schema from the one or more schemas of the first database based on (i) use of the keyword, the meaning or context of the keyword, and the NLQ and (ii) filtering a portion of the one or more schemas of the first database that is not related to the NLQ with the CoT prompt; (3) receiving the schema; and (4) obtaining, based on the schema, a statement specific to the telecommunications domain, at least one shot prompt, and a logical relation between columns in the schema; and (5) generating the system prompt based on the statement specific to the telecommunications domain, the at least one shot prompt, and the logical relation between columns in the schema.

[0198] In other embodiments, generating the system prompt in block 1606 includes calculating a similarity score between the keyword and at least one NLQ-SQL pair of at least one shot prompt. The at least one NLQ-SQL pair includes a sample NLQ that is a variation of the NLQ and is associated with a column of a table of the schema and the SQL corresponds to the sample NLQ. The generating (1604) in these embodiments further includes providing, to the LLM, an identity of a schema based on a logical relation between columns of at least a portion of at least one table from the knowledge graph.

[0199] In some embodiments, the NLQ-SQL pair is generated by the LLM.

[0200] The LLM may (i) generate at least one sample NLQ that is a variation of the NLQ for at least one column in the first database, (ii) take a column name and a description of the column from the knowledge graph, (iii) from the description, the LLM may create the at least one sample NLQ associated with the at least one column, and (iv) the LLM may pass, to a NLQ-to-SQL generator, the at least one sample NLQ and corresponding at least one column and column description to generate the SQL corresponding to the sample NLQ.

[0201] The NLQ-SQL pair may be used as a source to select the at least one shot prompt based on the NLQ.

[0202] Generating the system prompt in block 1606 may further include obtaining the logical statement of relation for the schema of the first database based on a connecting edge between at least two nodes in the knowledge graph.

[0203] Further, generating the system prompt in block 1606 may further include adding an additional one shot prompt to handle a specific case based on the NLQ and meaning or context of a keyword of the NLQ.

[0204] Generating the system prompt in block 1606 may further include receiving, from the LLM, a plurality of NLQ-SQL pairs for a plurality of few-shot examples. A respective NLQ-SQL pair from the plurality of NLQ-SQL pairs created in Figure 9, for example, may include a sample NLQ that is a variation of the NLQ, the sample NLQ may be associated with at least one column of at least one table of the first database, and the SQL may correspond to the sample NLQ.

[0205] In some embodiments, as shown in Figure 17, the method further optionally includes creating (block 1700) a vector database based on columns of tables of the data and descriptions of the columns; extracting (block 1702), from the knowledge graph, connector descriptions associated with the columns; extracting (block 1704) the plurality of NLQ-SQL pairs; and extracting (block 1706) statements specific to the telecommunications domain knowledge in the vector database.

[0206] Generating the system prompt in block 1606 may optionally include verifying the system prompt.

[0207] In some embodiments, the method optionally includes generating (block 1600) the knowledge graph based on one or more of an organization structure of the first database, the telecommunications domain knowledge, specifications of the telecommunications domain related to stored data in the database, and / or a description of requirements of the first database.

[0208] Generating the knowledge graph in block 1600 may include creating a description, including context, of at least some respective columns of a table in the first database; generating at least one of a statement of relation between nodes and a cause behind the relation, the nodes respectively including a table and at least one column of the table; identifying a relationship between columns from a plurality of columns in a plurality of tables; and identifying connections between the nodes.

[0209] The first database may include a plurality of tables based on the NLQ and a respective table may include a plurality of columns. In some embodiments, the method optionally includes generating (block 1602), for the first database, a description of at least one respective table in the plurality of tables and a description of at least some respective columns in the plurality of columns.

[0210] The information extracted may include one or more of a description of a table in a plurality of tables in the first database, a description of a column in a plurality of columns in the first database, information from the telecommunications domain knowledge, a specification document containing data in the database, and a requirement relating to a design of the first database.

[0211] The telecommunications domain knowledge may include a statement specific to the telecommunications domain based on the NLQ.

[0212] The relations within the data may include at least one of (i) a relation between a plurality of tables in the first database and how a first column of a first table in the plurality of tables is connected to at least a second column of at least a second table in the plurality of tables,(ii) a description of an entity relation based on the telecommunication domain knowledge, and (iii) a description of a relation in the relations.

[0213] The SQL generated by the LLM may be passed to a second database to extract data from the second database to generate a response to the NLQ. The second database may be a proprietary database, for example.

[0214] In some embodiments, the second database includes a relational database.

[0215] The extracted information may include text data indicating at least one or more phrases or sentences of a design document for the first database.

[0216] The knowledge graph may be generated based on an input including one or more of the following for the first database (i) a description of entities, a primary key and a foreign key per entity, and a relationship among the entities, (ii) a normalization of a design of the first database,(iii) an integrity rule of at least one entity, and (iv) indexing information of the entities.

[0217] The knowledge base may be reconfigurable in response to one or more changes in data including at least one of new information, new or changed user interactions, and changes in the telecommunications domain.

[0218] In some embodiments, the method optionally includes checking (block 1608) whether the system prompt includes sensitive data related to the telecommunications domain; when the system prompt includes sensitive data, marking (block 1610) the sensitive data; and performing (block 1612) at least one of masking the sensitive data or replacing the sensitive data with an alias before passing the system prompt to the LLM.

[0219] Operations of a computing device can be performed by the computing device 1900 of Figure 19. Operations of the computing device (implemented using the structure of Figure 19) have been discussed with reference to the flow charts of Figures 16 and 17 according to some embodiments of the present disclosure. Operations 1600, 1602, 1604 and / or 1608-1612 of Figure 16, and / or operations 1700-1706 of Figure 17, may be optional with respect to some embodiments of computing devices and related methods. Modules may be stored in memory 1905 of Figure 19, for example, and these modules may provide instructions so that when the instructions of a module are executed by respective computing device processor 1903 (also referred to herein as processing circuitry), computing device 1900 performs respective operations of the flow charts of Figures 16 and 17.

[0220] As shown in Figure 19, the computing device 1900 includes processor 1903 that is operatively coupled to memory 1905, network interface 1907, and / or any other component, or any combination thereof. Certain computing devices may utilize all or a subset of the components shown in Figure 19. The level of integration between the components may vary from one computing device to another computing device. Further, certain computing devices may contain multiple instances of a component, such as multiple processors, memories, etc.

[0221] The processor 1903 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory 1905. The processor 1903 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field- programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processor 1203 may include multiple central processing units (CPUs).

[0222] In the example, the network interface 1907 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and / or output devices. Examples of an output device include a display, a monitor, a printer, another output device, or any combination thereof. An input device may allow a user to capture information into the computing device 1900. Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presence-sensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, a force sensor, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device.

[0223] The memory 1905 may be or be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth. In one example, the memory 1905 includes one or more application programs, such as an operating system, web browser application, a widget, gadget engine, or other application,and corresponding data. The memory 1905 may store, for use by the computing device 1900, any of a variety of various operating systems or combinations of operating systems.

[0224] The memory 1905 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD-DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a USIM and / or ISIM, other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUICC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’ The memory 1905 may allow the computing device 1900 to access instructions, application programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory 1905 which may be or comprise a device-readable storage medium.

[0225] The processor 1903 may be configured to communicate with a network using the network interface 1907. The network interface 1907 may comprise one or more communication subsystems. The network interface 1907 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., another computing device, local computing device, edge node, cloud node, etc.). Each transceiver may include a transmitter and / or a receiver appropriate to provide network communications (e.g., optical, electrical, and so forth).

[0226] In the illustrated embodiment, communication functions of the network interface 1907 may include cellular communication, Wi-Fi communication, LPWAN communication, data communication, voice communication, multimedia communication, short-range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof. Communications may be implemented according to one or more communication protocols and / or standards, such as IEEE 802.11, Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LTE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol / intemet protocol (TCP / IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), QUIC, Hypertext Transfer Protocol (HTTP), and so forth.

[0227] Although the computing device described herein may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and / or software needed to perform the tasks, features, functions and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the computing device, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and / or the functionality of the components may be partitioned between the processor and the network interface. In another example, non- computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.

[0228] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer- readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer-readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally.

[0229] In certain embodiments, a computing device (1900) is provided. The computing device (1900) includes at least one processor (1903); and at least one memory (1905) connected to the at least one processor (1903) and storing program code that is executed by the at least one processor to perform operations. The operations include to perform some or all of the functionality described herein.

[0230] In certain embodiments, a computer program product is provided including a non- transitory storage medium (1905) including program code (1909) to be executed by at least one processor (1903) of a computing device (1900). Execution of the program code causes the computing device to perform operations. The operations include to perform some or all of the functionality described herein.

[0231] Figure 18 shows an example of a communication system 1800 in accordance with some embodiments.

[0232] In the example, the communication system 1800 includes a telecommunication network 1802 that includes an access network 1804, such as a RAN, and a core network 1806, which includes one or more core network nodes 1808. The access network 1804 includes one or more access network nodes, such as network nodes 1810a and 1810b (one or more of which may be generally referred to as network nodes 1810), or any other similar 3GPP access node or non- 3 GPP access point. The network nodes 1810 facilitate direct or indirect connection of user equipment (UE) (also referred to herein as “user device” or a “communication device”), such as by connecting UEs 1812a, 1812b, 1812c, and 1812d (one or more of which may be generally referred to as UEs 1812) to the core network 1806 over one or more wireless connections. A computing device (e.g., computing device 1900) can be a network node 1810 or another node comprising a computing device 1900 in communication system 1800.

[0233] Example wireless communications over a wireless connection include transmitting and / or receiving wireless signals using electromagnetic waves, radio waves, infrared waves, and / or other types of signals suitable for conveying information without the use of wires, cables, or other material conductors. Moreover, in different embodiments, the communication system 1800 may include any number of wired or wireless networks, network nodes, UEs, and / or any other components or systems that may facilitate or participate in the communication of data and / or signals whether via wired or wireless connections. The communication system 1300 may include and / or interface with any type of communication, telecommunication, data, cellular, radio network, and / or other similar type of system.

[0234] The UEs 1812 may be any of a wide variety of communication devices, including wireless devices arranged, configured, and / or operable to communicate wirelessly with the network nodes 1810 and other communication devices. Similarly, the network nodes 1810 are arranged, capable, configured, and / or operable to communicate directly or indirectly with the UEs 1812 and / or with other network nodes or equipment in the telecommunication network 1802 to enable and / or provide network access, such as wireless network access, and / or to perform other functions, such as administration in the telecommunication network 1802.

[0235] In the depicted example, the core network 1806 connects the network nodes 1810 to one or more hosts, such as host 1816. These connections may be direct or indirect via one or more intermediary networks or devices. In other examples, network nodes may be directly coupled to hosts. The core network 1806 includes one more core network nodes (e.g., core network node 1808) that are structured with hardware and software components. Features of these components may be substantially similar to those described with respect to the UEs, network nodes, and / or hosts, such that the descriptions thereof are generally applicable to the corresponding components of the core network node 1808. Example core network nodes include functions of one or more of a Mobile Switching Center (MSC), Mobility Management Entity (MME), Home Subscriber Server (HSS), Access and Mobility Management Function (AMF), Session Management Function (SMF), Authentication Server Function (AUSF), Subscription Identifier De-concealing function (SIDF), Unified Data Management (UDM), Security Edge Protection Proxy (SEPP), Network Exposure Function (NEF), and / or a User Plane Function (UPF).

[0236] The host 1816 may be under the ownership or control of a service provider other than an operator or provider of the access network 1804 and / or the telecommunication network 1802, and may be operated by the service provider or on behalf of the service provider. The host 1816 may host a variety of applications to provide one or more service. Examples of such applications include live and pre-recorded audio / video content, data collection services such as retrieving and compiling data on various ambient conditions detected by a plurality of UEs, analytics functionality, social media, functions for controlling or otherwise interacting with remote devices, functions for an alarm and surveillance center, or any other such function performed by a server.

[0237] As a whole, the communication system 1800 of Figure 18 enables connectivity between the UEs, network nodes, and hosts. In that sense, the communication system may be configured to operate according to predefined rules or procedures, such as specific standards that include, but are not limited to: Global System for Mobile Communications (GSM); Universal Mobile Telecommunications System (UMTS); Long Term Evolution (LTE), and / or other suitable 2G, 3G, 4G, 5G standards, or any applicable future generation standard (e.g., 6G); wireless local area network (WLAN) standards, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (WiFi); and / or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Microwave Access (WiMax), Bluetooth, Z-Wave, Near Field Communication (NFC) ZigBee, LiFi, and / or any low-power wide-area network (LPWAN) standards such as LoRa and Sigfox.

[0238] In some examples, the telecommunication network 1802 is a cellular network that implements 3GPP standardized features. Accordingly, the telecommunications network 1802 maysupport network slicing to provide different logical networks to different devices that are connected to the telecommunication network 1802. For example, the telecommunications network 1802 may provide URLLC services to some UEs, while providing eMBB services to other UEs, and / or mMTC / Massive loT services to yet further UEs.

[0239] In some examples, the UEs 1812 are configured to transmit and / or receive information without direct human interaction. For instance, a UE may be designed to transmit information to the access network 1804 on a predetermined schedule, when triggered by an internal or external event, or in response to requests from the access network 1804. Additionally, a UE may be configured for operating in single- or multi-RAT or multi-standard mode. For example, a UE may operate with any one or combination of Wi-Fi, NR and LTE, i.e. being configured for multi -radio dual connectivity (MR-DC), such as E-UTRAN (Evolved-UMTS Terrestrial Radio Access Network) New Radio - Dual Connectivity (EN-DC).

[0240] In the example, the hub 1814 communicates with the access network 1804 to facilitate indirect communication between one or more UEs (e.g., UE 1812c and / or 1812d) and network nodes (e.g., network node 1810b). In some examples, the hub 1814 may be a controller, router, content source and analytics, or any of the other communication devices described herein regarding UEs. For example, the hub 1814 may be a broadband router enabling access to the core network 1806 for the UEs. As another example, the hub 1814 may be a controller that sends commands or instructions to one or more actuators in the UEs. Commands or instructions may be received from the UEs, network nodes 1810, or by executable code, script, process, or other instructions in the hub 1814. As another example, the hub 1814 may be a data collector that acts as temporary storage for UE data and, in some embodiments, may perform analysis or other processing of the data. As another example, the hub 1814 may be a content source. For example, for a UE that is a VR headset, display, loudspeaker or other media delivery device, the hub 1814 may retrieve VR assets, video, audio, or other media or data related to sensory information via a network node, which the hub 1814 then provides to the UE either directly, after performing local processing, and / or after adding additional local content. In still another example, the hub 1814 acts as a proxy server or orchestrator for the UEs, in particular if one or more of the UEs are low energy loT devices.

[0241] The hub 1814 may have a constant / persistent or intermittent connection to the network node 1810b. The hub 1814 may also allow for a different communication scheme and / or schedule between the hub 1814 and UEs (e.g., UE 1812c and / or 1812d), and between the hub 1814 and the core network 1806. In other examples, the hub 1814 is connected to the core network 1806 and / or one or more UEs via a wired connection. Moreover, the hub 1814 may be configured to connectto an M2M service provider over the access network 1804 and / or to another UE over a direct connection. In some scenarios, UEs may establish a wireless connection with the network nodes 1810 while still connected via the hub 1814 via a wired or wireless connection. In some embodiments, the hub 1814 may be a dedicated hub - that is, a hub whose primary function is to route communications to / from the UEs from / to the network node 1810b. In other embodiments, the hub 1814 may be a non-dedicated hub - that is, a device which is capable of operating to route communications between the UEs and network node 1810b, but which is additionally capable of operating as a communication start and / or end point for certain data channels.

[0242] Further definitions and embodiments are discussed below.

[0243] In the above-description of certain embodiments of the present disclosure, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which concepts of the present disclosure belong. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0244] When an element is referred to as being “connected”, “coupled”, “responsive”, or variants thereof to another element, it can be directly connected, coupled, or responsive to the other element or intervening elements may be present. In contrast, when an element is referred to as being “directly connected”, “directly coupled”, “directly responsive”, or variants thereof to another element, there are no intervening elements present. Like numbers refer to like elements throughout. Furthermore, “coupled”, “connected”, “responsive”, or variants thereof as used herein may include wirelessly coupled, connected, or responsive. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Well-known functions or constructions may not be described in detail for brevity and / or clarity. The term “and / or” (abbreviated “ / ”) includes any and all combinations of one or more of the associated listed items.

[0245] It will be understood that although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Thus a first element / operation in some embodiments could be termed a second element / operation in other embodiments without departing from the teachings of concepts of thepresent disclosure. The same reference numerals or the same reference designators denote the same or similar elements throughout the specification.

[0246] As used herein, the terms “comprise”, “comprising”, “comprises”, “include”, “including”, “includes”, “have”, “has”, “having”, or variants thereof are open-ended, and include one or more stated features, integers, elements, steps, components, or functions but does not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions, or groups thereof. Furthermore, as used herein, the common abbreviation “e.g.”, which derives from the Latin phrase “exempli gratia,” may be used to introduce or specify a general example or examples of a previously mentioned item, and is not intended to be limiting of such item. The common abbreviation “i.e.”, which derives from the Latin phrase “id est,” may be used to specify a particular item from a more general recitation.

[0247] Example embodiments are described herein with reference to block diagrams and / or flowchart illustrations of computer-implemented methods, apparatus (systems and / or devices) and / or computer program products. It is understood that a block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by computer program instructions that are performed by one or more computer circuits. These computer program instructions may be provided to a processor circuit of a general purpose computer circuit, special purpose computer circuit, and / or other programmable data processing circuit to produce a machine, such that the instructions, which execute via the processor of the computer and / or other programmable data processing apparatus, transform and control transistors, values stored in memory locations, and other hardware components within such circuitry to implement the functions / acts specified in the block diagrams and / or flowchart block or blocks, and thereby create means (functionality) and / or structure for implementing the functions / acts specified in the block diagrams and / or flowchart block(s).

[0248] These computer program instructions may also be stored in a tangible computer- readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the functions / acts specified in the block diagrams and / or flowchart block or blocks. Accordingly, embodiments of the present disclosure may be embodied in hardware and / or in software (including firmware, resident software, micro-code, etc.) that runs on a processor such as a digital signal processor, which may collectively be referred to as “circuitry,” “a module” or variants thereof.

[0249] It should also be noted that in some alternate implementations, the functions / acts noted in the blocks may occur out of the order noted in the flowcharts. For example, two blocks shownin succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality / acts involved. Moreover, the functionality of a given block of the flowcharts and / or block diagrams may be separated into multiple blocks and / or the functionality of two or more blocks of the flowcharts and / or block diagrams may be at least partially integrated. Finally, other blocks may be added / inserted between the blocks that are illustrated, and / or blocks / operations may be omitted without departing from the scope of the present disclosure. Moreover, although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows.

[0250] Many variations and modifications can be made to the embodiments without substantially departing from the principles of the present disclosure. All such variations and modifications are intended to be included herein within the scope of present disclosure. Accordingly, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the examples of embodiments are intended to cover all such modifications, enhancements, and other embodiments, which fall within the spirit and scope of the present disclosure. Thus, to the maximum extent allowed by law, the scope of the present disclosure is to be determined by the broadest permissible interpretation of the present disclosure including the examples of embodiments and their equivalents, and shall not be restricted or limited by the foregoing detailed description.

Claims

Claims:

1. A computer-implemented method of generating a system prompt for a large language model, LLM, to convert a natural language query, NLQ, to a structured query language, SQL, the method comprising: generating (1606) the system prompt for the NLQ based on information extracted from a first database, the information identified from a knowledge graph comprising a structured view of data comprising telecommunications domain knowledge and relations within the data filtered by a chain of thought prompt that represents a logical relation between a plurality of nodes in the knowledge graph; and passing (1614) the system prompt to the LLM to convert the NLQ to the SQL.

2. The computer-implemented method of Claim 1, wherein the LLM uses the at least one logical relation to prepare at least one statement of logical relation and uses the at least one statement of logical relation to prepare the chain of thought prompt, and further comprising: receiving (1604) the chain of though prompt.

3. The computer-implemented method of any one of Claims 1 or 2, wherein the first database comprises one or more schemas respectively comprising one or more tables comprising a column.

4. The computer-implemented method of any one of Claims 1 to 3, wherein generating (1604) the system prompt comprises receiving a keyword and a meaning or context of the keyword extracted from the NLQ; finding a schema from the one or more schemas of the first database based on (i) use of the keyword, the meaning or context of the keyword, and the NLQ and (ii) filtering a portion of the one or more schemas of the first database that is not related to the NLQ with the chain of thought prompt; receiving the schema; obtaining, based on the schema, a statement specific to the telecommunications domain, at least one shot prompt, and a logical relation between columns in the schema; and generating the system prompt based on the statement specific to the telecommunications domain, the at least one shot prompt, and the logical relation between columns in the schema.

5. The computer-implemented method of any one of Claims 1 to 4, wherein generating (1606) the system prompt comprises calculating a similarity score between the keyword and at least one NLQ-SQL pair of at least one shot prompt, the at least one NLQ-SQL pair comprising a sample NLQ that is a variation of the NLQ and is associated with a column of a table of the schema and the SQL corresponds to the sample NLQ; and providing, to the LLM, an identity of a schema based on a logical relation between columns of at least a portion of at least one table from the knowledge graph.

6. The computer-implemented method of Claim 5, wherein the NLQ-SQL pair is generated by the LLM.

7. The computer-implemented method of Claim 6, wherein the LLM (i) generates at least one sample NLQ that is a variation of the NLQ for at least one column in the first database,(ii) the LLM takes a column name and a description of the column from the knowledge graph,(iii) from the description, the LLM creates the at least one sample NLQ associated with the at least one column, and (iv) the LLM passes, to a NLQ-to-SQL generator, the at least one sample NLQ and corresponding at least one column and column description to generate the SQL corresponding to the sample NLQ.

8. The computer-implemented method of any one of Claims 5 to 7, wherein the NLQ- SQL pair is used as a source to select the at least one shot prompt based on the NLQ.

9. The computer-implemented method of any one of Claims 4 to 6, wherein generating (1606) the system prompt further comprises obtaining the logical statement of relation for the schema of the first database based on a connecting edge between at least two nodes in the knowledge graph.

10. The computer-implemented method of any one of Claims 4 to 9, wherein generating (1606) the system prompt further comprises adding an additional shot prompt to handle a specific case based on the NLQ and meaning or context of a keyword of the NLQ.

11. The computer-implemented method of any one of Claims 4 to 10, wherein generating (1606) the system prompt further comprises verifying the system prompt.

12. The computer-implemented method of any one of Claims 1 to 11, further comprising: generating (1600) the knowledge graph based on one or more of an organization structure of the first database, the telecommunications domain knowledge, specifications of the telecommunications domain related to stored data in the database, and / or a description of requirements of the first database.

13. The computer-implemented method of any one of Claims 1 to 12, wherein the first database comprises a plurality of tables based on the NLQ and a respective table comprises a plurality of columns.

14. The computer-implemented method of Claim 13, further comprising: generating (1602), for the first database, a description of at least one respective table in the plurality of tables and a description of at least some respective columns in the plurality of columns.

15. The computer-implemented method of any one of Claims 1 to 14, wherein the information extracted comprises one or more of a description of a table in a plurality of tables in the first database, a description of a column in a plurality of columns in the first database, information from the telecommunications domain knowledge, a specification document containing data in the database, and a requirement relating to a design of the first database.

16. The computer-implemented method of any one of Claims 1 to 15, wherein the telecommunications domain knowledge comprises a statement specific to the telecommunications domain based on the NLQ.

17. The computer-implemented method of any one of Claims 1 to 16, wherein the relations within the data comprise at least one of (i) a relation between a plurality of tables in the first database and how a first column of a first table in the plurality of tables is connected to at least a second column of at least a second table in the plurality of tables, (ii) a description of an entity relation based on the telecommunication domain knowledge, and (iii) a description of a relation in the relations.

18. The computer-implemented method of any one of Claims 1 to 17, wherein generating (1600) the knowledge graph comprises creating a description, including context, of at least some respective columns of a table in the first database; generating at least one of a statement of relation between nodes and a cause behind the relation, the nodes respectively comprising a table and at least one column of the table; identifying a relationship between columns from a plurality of columns in a plurality of tables; and identifying connections between the nodes.

19. The computer-implemented method of any of Claims 1 to 18, wherein the SQL generated by the LLM is passed to a second database to extract data from the second database to generate a response to the NLQ.

20. The computer-implemented method of any one of Claims 1 to 19, wherein the second database comprises a relational database.

21. The computer-implemented method of any one of Claims 1 to 20, wherein the extracted information comprises text data indicating at least one or more phrases or sentences of a design document for the first database.

22. The computer-implemented method of any one of Claims 1 to 21, wherein the knowledge graph is generated based on an input comprising one or more of the following for the first database (i) a description of entities, a primary key and a foreign key per entity, and a relationship among the entities, (ii) a normalization of a design of the first database, (iii) an integrity rule of at least one entity, and (iv) indexing information of the entities.

23. The computer-implemented method of any one of Claims 1 to 22, wherein the knowledge base is reconfigurable in response to one or more changes in data comprising at least one of new information, new or changed user interactions, and changes in the telecommunications domain.

24. The computer-implemented method of any one of Claims 1 to 23, further comprising:checking (1608) whether the system prompt comprises sensitive data related to the telecommunications domain; when the system prompt comprises sensitive data, marking (1610) the sensitive data; and performing (1612) at least one of masking the sensitive data or replacing the sensitive data with an alias before passing the system prompt to the LLM.

25. The computer-implemented method of any one of Claims 1 to 24, wherein generating (1604) the system prompt comprises further comprises receiving, from the LLM, a plurality of NLQ-SQL pairs for a plurality of few-shot examples, wherein a respective NLQ- SQL pair from the plurality of NLQ-SQL pairs comprises a sample NLQ that is a variation of the NLQ, the sample NLQ is associated with at least one column of at least one table of the first database, and the SQL corresponds to the sample NLQ.

26. The computer-implemented method of Claim 25, the method further comprising: creating (1700) a vector database based on columns of tables of the data and descriptions of the columns; extracting (1702), from the knowledge graph, connector descriptions associated with the columns; extracting (1704) the plurality of NLQ-SQL pairs; and extracting (1706) statements specific to the telecommunications domain knowledge and in the vector database.

27. The computer-implemented method of any one of Claims 1 to 26, wherein the chain of thought comprises a plurality of logical statements of relations that represents a complex selected database schema comprising a plurality of tables and complex relations between columns of the plurality of tables, the chain of thought is included in the system prompt based on the complexity of the NLQ and the complex selected database schema.

28. A computing device (1900) configured to generate a system prompt for a large language model, LLM, to convert a natural language query, NLQ, to a structured query language, SQL, the computing device comprising: at least one processor (1903);at least one memory (1905) connected to the at least one processor (1903) and storing program code (1909) that is executed by the at least one processor to perform operations comprising: generate the system prompt for the NLQ based on information extracted from a first database, the information identified from a knowledge graph comprising a structured view of data comprising telecommunications domain knowledge and relations within the data filtered by a chain of thought prompt that represents a logical relation between a plurality of nodes in the knowledge graph; and pass the system prompt to the LLM to convert the NLQ to the SQL.

29. The computing device of Claim 28, wherein the at least one memory (1905) is connected to the at least one processor (1903) and stores program code that is executed by the at least one processor to perform operations according to any one of Claims 2 to 27.

30. A computer program product comprising a non-transitory storage medium (1905) including program code (1909) to be executed by at least one processor (1903) of a computing device (1900) configured to generate a system prompt for a large language model, LLM, to convert a natural language query, NLQ, to a structured query language, SQL, whereby execution of the program code causes the computing device to perform operations comprising: generate the system prompt for the NLQ based on information extracted from a first database, the information identified from a knowledge graph comprising a structured view of data comprising telecommunications domain knowledge and relations within the data filtered by a chain of thought prompt that represents a logical relation between a plurality of nodes in the knowledge graph; and pass the system prompt to the LLM to convert the NLQ to the SQL.

31. The computer program product of Claim 30, whereby execution of the program code causes the computing device to perform operations according to any of Claims 2 to 27.

Images