Computer implementation methods, computer systems, and computer programs (natural language query processing using asset analysis based on large-scale language models)

Abstract

This invention provides a computer implementation method that includes processing of natural language queries using a large-scale language model (LLM) for analyzing and querying databases. [Solution] The method uses an LLM to generate a first SQL statement based on a user query provided by the user, extracts multiple entities based on the first SQL statement, performs a semantic search on indexes of multiple databases using the extracted entities, determines the confidence score of each of the multiple databases, selects a database based on the confidence score, extracts the schema of the selected database, generates a second SQL statement using the LLM based on the user query and schema, executes the second SQL statement on the selected database, and generates query results presented to the user.

Description

Technical Field

[0001] Embodiments of the present invention relate to natural language query processing, and more specifically, to the processing of natural language queries using a large language model for analyzing and querying a database.

Background Art

[0002] In the field of natural language processing, a user's natural language query can be processed to generate a response. Depending on the query, a structured database may store data relevant to answering the user's query. However, without an indication of which database to reference, it can be difficult to identify structured data that can answer the user's query.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Without an indication of which database to reference, it can be difficult to identify structured data that can answer the user's query.

Means for Solving the Problems

[0004] According to one embodiment of the present invention, a system for processing user queries is provided. Using a large language model (LLM), a first Structured Query Language (SQL) statement is generated based on a user query provided by the user. Based on the first SQL statement, multiple entities, including one or more columns, filter columns, and category filters, are extracted. Using one or more columns, filter columns, and category filters, a semantic search is performed against indexes of multiple databases to determine a confidence score for each of the multiple databases. This confidence score indicates the degree of match for each database with respect to one or more extracted columns, filter columns, and category filters. A database is selected based on the confidence score, and the schema of the selected database is extracted. Using the LLM, a second SQL statement is generated based on the user query and schema. The second SQL statement is executed on the selected database, and query results are generated. The query results are presented to the user. Embodiments of the present invention further include methods and computer program products for processing user queries in substantially the same manner as described above.

[0005] Various other embodiments of the present invention will now be discussed. These various embodiments are optional and should not be construed as necessary to carry out any aspect of the embodiments presented herein.

[0006] In some embodiments, the selected database is validated using the LLM by providing the LLM with prompts including a user query, the schema of the selected database, and a request to determine whether the selected database contains data that supports the user query. The embodiments presented herein ensure that the database is capable of supporting the user query by validating the database. Thus, processing requirements are reduced by avoiding querying databases that are not expected to provide results relevant to the user query.

[0007] Candidate databases may be ranked using a confidence score indicating the degree of match for each candidate database with respect to entities (e.g., columns, filter columns, and / or categorical filters) extracted from the database query initially generated by the LLM based on the user query. In some embodiments, the confidence score is determined by evaluating the column names of each database with respect to one or more extracted columns or filter columns. The accuracy of the results is improved by determining that the column names of the candidate databases are relevant to the user query.

[0008] In some embodiments, a confidence score is determined by generating synonyms for one or more extracted columns or filter columns and evaluating the column names of each database with respect to the generated synonyms. The use of synonyms allows for the identification of additional relevant databases, thereby expanding the set of candidate databases and consequently improving the accuracy of query processing.

[0009] In some embodiments, a confidence score is determined by evaluating one or more extracted category filters against the category filters of each database. By analyzing the database against the category filters, embodiments of the present invention ensure that the database stores the types of data relevant to user queries. Thus, the accuracy of the results presented to the user is improved.

[0010] In some embodiments, the LLM is provided with prompts to generate SQL statements that include one or more examples of desired output. One-shot or multi-shot prompts improve the LLM's output, thereby improving the analysis of candidate databases and the processing of selected databases by generating more accurate database queries.

[0011] In some embodiments, the LLM is provided with prompts to generate SQL statements that conform to a specific SQL dialect. This improves the accuracy of database query processing by ensuring that the SQL statements match the dialect of the selected database. [Brief explanation of the drawing]

[0012] Generally, similar reference numbers are used to specify similar components in various diagrams.

[0013] [Figure 1] This is a schematic diagram illustrating an example of a computing environment according to one embodiment of the present invention.

[0014] [Figure 2] This is a block diagram of query processing code according to one embodiment of the present invention.

[0015] [Figure 3] This is a flowchart illustrating a process for analyzing a database and generating database queries, according to one embodiment of the present invention.

[0016] [Figure 4] This is a flowchart illustrating a method for executing user queries according to one embodiment of the present invention. [Modes for carrying out the invention]

[0017] Embodiments of the present invention relate to natural language query processing, and more specifically, to the processing of natural language queries using large-scale language models for analyzing and querying databases. In the fields of natural language processing (NLP) and machine learning (ML), users can submit natural language queries for processing by NLP / ML models. For example, if a user wants to find information about the best-selling products in various cities, the user might submit a query such as "What are the best-selling products in major US cities?". To respond to such a query, relational databases may be searched to identify results. However, without instructions on which specific databases to search, it may not be possible to accurately find the structured data relevant to the answer to the user's question.

[0018] The embodiments presented herein address this challenge by employing a multi-stage approach. This approach first identifies a specific database to be used when responding to a user query, automatically generates a database query to that specific database, and obtains results usable in answering the user query. Specifically, a large-scale language model is used to generate the database query based on the user query. The database query can take the form of any standardized language, such as Standard Query Language (SQL). Next, entities included in the database query may be extracted and used to identify the relevant database to be used when responding to the user query. Specifically, these entities may include columns, filter columns, and / or category filters. These entities may be used to perform a semantic search using a search engine that indexes multiple databases. In these databases, each database is scored for its relevance to the user query. Once the relevant database is identified, the schema of the relevant database may be used by the large-scale language model to generate an improved database query. This query may then be executed against the relevant database to generate a response to the user query.

[0019] Thus, embodiments of the present invention provide a novel approach to natural language processing. In this approach, a generative machine learning model generates database queries to identify specific databases that are highly relevant to a user query, and then uses the identified databases to generate a response to the user query. By using a unique data structure indexed across various databases, the embodiments presented herein enable rapid ranking of databases in terms of relevance to the user query, thereby identifying and querying the most relevant database. Therefore, embodiments of the present invention improve the field of natural language processing and query processing by using a centralized search that queries only relevant databases. This approach reduces processing, memory, and network resources by leveraging queries in a way that retrieves only relevant data. Furthermore, by using indexes for database selection, computing resource requirements are further reduced because there is no need to access the underlying database until the database is determined to be relevant. Thus, embodiments provide a practical application for improving the relevance and accuracy of NLP / ML-generated responses to user queries.

[0020] In some embodiments, the selected database is validated using the LLM by providing the LLM with prompts including a user query, the schema of the selected database, and a request to determine whether the selected database contains data that supports the user query. The embodiments presented herein ensure that the database is capable of supporting the user query by validating the database. Thus, processing requirements are reduced by avoiding querying databases that are not expected to provide results relevant to the user query.

[0021] Candidate databases may be ranked using a confidence score indicating the degree of match for each candidate database with respect to entities (e.g., columns, filter columns, and / or category filters) extracted from a database query initially generated by an LLM based on a user query. In some embodiments, the confidence score is determined by evaluating the column names of each database with respect to one or more of the extracted columns or filter columns. By determining that the column names of the candidate databases are relevant to the user query, the accuracy of the results is improved.

[0022] In some embodiments, synonyms for one or more of the extracted columns or filter columns are generated, and the confidence score is determined by evaluating the column names of each database with respect to the generated synonyms. By using synonyms, additional relevant databases can be identified, thereby expanding the set of candidate databases and improving the accuracy of query processing.

[0023] In some embodiments, the confidence score is determined by evaluating one or more of the extracted category filters with respect to the category filters of each database. By analyzing the database with respect to the category filters, embodiments of the present invention ensure that the database stores data of the type relevant to the user query. Thus, the accuracy of the results presented to the user is improved.

[0024] In some embodiments, a prompt for generating an SQL statement including one or more examples of a desired output is provided to the LLM. The output of the LLM is improved by a one-shot prompt or a multi-shot prompt, thereby generating a more accurate database query, which improves the analysis of candidate databases and the processing of the selected database.

[0025] In some embodiments, a prompt for generating an SQL statement compliant with a particular SQL dialect is provided to the LLM. In this way, the accuracy of database query processing is improved by ensuring that the SQL statement exactly matches the dialect of the selected database.

[0026] Various aspects of the present disclosure are illustrated by descriptions, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of a computer program product (CPP). For any flowchart, depending on the technology involved, operations may be performed in an order different from that shown in a given flowchart. For example, again depending on the technology involved, two operations shown in consecutive blocks of a flowchart may be performed in reverse order, simultaneously, or at least partially overlapping in time, as a single integrated step.

[0027] Embodiments of a computer program product ("CPP Embodiment" or "CPP") are terms used in this disclosure to describe any set of one or more storage media ("Multiple Media") that together comprise a set of one or more storage devices containing machine-readable code corresponding to instructions and / or data for performing computer operations specified in a given CPP claim. "Storage device" is any tangible device capable of holding and storing instructions for use by a computer processor. Computer-readable storage media may, but are not limited to, electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, mechanical storage media, or any suitable combination of the media described above. Some known types of storage devices, including these media, include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices (e.g., punch cards, or pits / lands formed on the main surface of a disk), or any suitable combination of the aforementioned media. When the term "computer-readable storage medium" is used in this disclosure, it should not be interpreted as storage in the form of transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, optical pulses passing through optical fiber cables, electrical signals transmitted through wires, and / or other transmission media.As will be understood by those skilled in the art, data is typically moved at several intermittent points during the normal operation of a storage device, such as during access, defragmentation, or garbage collection. However, this does not mean that the storage device is temporary, because data is not temporary while it is stored there.

[0028] Referring to Figure 1, the computing environment 100 includes an example of an environment for executing at least a portion of the computer code (such as query processing code 200) involved in carrying out the method of the present invention. In addition to block 200, the computing environment 100 includes, for example, a computer 101, a wide area network (WAN) 102, an end user device (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, the computer 101 includes a processor set 110 (including processing circuits 120 and a cache 121), a communication fabric 111, volatile memory 112, persistent storage 113 (including the operating system 122 and block 200 as shown above), a peripheral device set 114 (including a user interface (UI) device set 123, storage 124, and an Internet of Things (IoT) sensor set 125), and a network module 115. The remote server 104 includes a remote database 130. The public cloud 105 includes a gateway 140, a cloud orchestration module 141, a host physical machine set 142, a virtual machine set 143, and a container set 144.

[0029] Computer 101 may take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch or other wearable computer, mainframe computer, quantum computer, or other known or future-developed forms of computer or mobile device capable of running programs, accessing networks, or querying databases such as remote database 130. As is well understood in the field of computer technology, and depending on the technology, the implementation of a computer implementation method may be distributed across multiple computers and / or multiple locations. On the other hand, in this description of the computing environment 100, in order to keep the description as concise as possible, the detailed description focuses on a single computer, specifically computer 101. Computer 101 may be located in the cloud even if it is not shown in the cloud in Figure 1. On the other hand, computer 101 does not need to be located in the cloud, except to the extent that it can be definitively shown.

[0030] The processor set 110 includes one or more computer processors of any kind currently known or to be developed in the future. The processing circuitry 120 may be distributed across multiple packages, for example, multiple interconnected integrated circuit chips. The processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. The cache 121 is memory located within the processor chip package and is typically used for data or code that should be available for high-speed access by threads or cores operating on the processor set 110. The cache memory is typically divided into multiple levels depending on its relative proximity to the processing circuitry. Alternatively, some or all of the processor set's cache may be located "off-chip". In some computing environments, the processor set 110 may be designed to handle qubits and perform quantum computing.

[0031] Computer-readable program instructions are typically loaded onto computer 101 and cause the processor set 110 of computer 101 to perform a series of operational steps, thereby realizing the computer implementation method. As a result, the instructions thus executed instantiate the methods specified in the flowcharts and / or descriptions of the computer implementation methods contained herein (collectively referred to as the "Methods of the Invention"). These computer-readable program instructions are stored in various types of computer-readable storage media, such as the cache 121 and other storage media described later. The program instructions and associated data are accessed by the processor set 110 to control and direct the implementation of the Methods of the Invention. In the computing environment 100, at least some of the instructions for implementing the Methods of the Invention may be stored in blocks 200 within persistent storage 113.

[0032] The communication fabric 111 is a signal transmission path that allows various components of the computer 101 to communicate with each other. Typically, this fabric is made up of switches and conductive paths, such as buses, bridges, physical input / output ports, and switches and conductive paths that make up such components. Other types of signal communication paths, such as optical fiber communication paths and / or wireless communication paths, may also be used.

[0033] Volatile memory 112 refers to any type of volatile memory currently known or to be developed in the future. Examples include dynamic random-access memory (RAM) or static RAM. Typically, volatile memory 112 is characterized by random access, but this is not required unless explicitly stated. In computer 101, the volatile memory 112 is located in a single package and resides inside computer 101, but alternatively or additionally, the volatile memory may be distributed across multiple packages and / or located externally to computer 101.

[0034] The persistent storage 113 is any form of non-volatile storage for a computer, currently known or to be developed in the future. The non-volatility of this storage means that the stored data is maintained regardless of whether power is supplied to the computer 101 and / or directly to the persistent storage 113. The persistent storage 113 may be read-only memory (ROM), but typically at least a portion of the persistent storage is writable, deletable, and rewritable. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. The operating system 122 can take several forms, including various known proprietary operating systems using a kernel, or open-source POSIX (Portable Operating System Interface) type operating systems. The code contained in block 200 typically includes at least a portion of computer code involved in carrying out the method of the present invention.

[0035] The peripheral device set 114 includes a set of peripheral devices for the computer 101. Data communication connections between the computer 101's peripheral devices and other components may be implemented in various ways, including Bluetooth® connections, Near-Field Communication (NFC) connections, connections made by cables (e.g., Universal Serial Bus (USB) type cables), insert-type connections (e.g., Secure Digital (SD) cards), connections made via local area communication networks, and even connections made via wide area networks such as the Internet. In various embodiments, the UI device set 123 may include components such as a display screen, speakers, microphones, wearable devices (e.g., goggles and smartwatches), keyboards, mice, printers, touchpads, game controllers, and haptic devices. Storage 124 is external storage such as an external hard drive, or insertable storage such as an SD card. Storage 124 may be persistent and / or volatile. In some embodiments, storage 124 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where computer 101 needs to have large-capacity storage (for example, computer 101 locally stores and manages a large database), this storage may be provided by peripheral storage devices designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor set 125 consists of sensors that can be used in Internet of Things applications. For example, one sensor may be a thermometer and another may be a motion detector.

[0036] The network module 115 is a collection of computer software, hardware, and firmware that enables computer 101 to communicate with other computers via the WAN 102. The network module 115 may include hardware such as a modem or Wi-Fi® signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or web browser software for exchanging data over the Internet. In some embodiments, the network control and network forwarding functions of the network module 115 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing software-defined networking (SDN)), the control and forwarding functions of the network module 115 are performed on physically separate devices, resulting in the control function managing multiple different network hardware devices. Computer-readable program instructions for implementing the method of the present invention can typically be downloaded from an external computer or external storage device to computer 101 via a network adapter card or network interface included in the network module 115.

[0037] WAN102 is any wide area network (e.g., the Internet) capable of exchanging computer data over non-local distances using any currently known or future-developed technologies for exchanging computer data. In some embodiments, WAN102 may be replaced and / or complemented by a local area network (LAN), such as a Wi-Fi network, designed to exchange data between devices located in the local area. WANs and / or LANs typically include computer hardware such as copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0038] The end-user device (EUD) 103 is any computer system used and controlled by an end-user (e.g., a customer of the company operating computer 101) and can take any of the forms described above in relation to computer 101. The EUD 103 typically receives useful and beneficial data from the operation of computer 101. For example, in a hypothetical case where computer 101 is designed to provide recommendations to the end-user, these recommendations would typically be transmitted from computer 101's network module 115 to the EUD 103 via the WAN 102. In this way, the EUD 103 can display or otherwise present the recommendations to the end-user. In some embodiments, the EUD 103 may be a client device, such as a thin client, heavy client, mainframe computer, or desktop computer.

[0039] The remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. The remote server 104 may be controlled and used by the same entity that operates computer 101. The remote server 104 represents a machine that collects and stores useful and beneficial data for use by other computers, such as computer 101. For example, in a hypothetical case where computer 101 is designed and programmed to make recommendations based on historical data, this historical data may be provided to computer 101 from the remote database 130 of the remote server 104.

[0040] The public cloud 105 is an arbitrary computer system available for use by multiple entities, providing on-demand availability of computer system resources and / or other computer functions, particularly data storage (cloud storage) and computing power, without requiring direct, voluntary management by the user. Cloud computing typically leverages resource sharing to achieve consistency and economies of scale. Direct and voluntary management of the computing resources of the public cloud 105 is performed by the computer hardware and / or software of the cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by a virtual computing environment running on various computers that make up the host physical machine set 142. The host physical machine set refers to all physical computers within and / or available to the public cloud 105. The virtual computing environment (VCE) typically takes the form of virtual machines in the virtual machine set 143 and / or containers in the container set 144. These VCEs may be stored as images and are understood to be transferable either as images or after VCE instantiation between various physical machine hosts. The cloud orchestration module 141 manages the transfer and storage of images, deploys new VCE instantiations, and manages active instances resulting from VCE deployments. The gateway 140 is a collection of computer software, hardware, and firmware that enables the public cloud 105 to communicate over the WAN 102.

[0041] Here, we will provide some further explanation of virtual computing environments (VCEs). A VCE can be stored as an "image." From that image, a new active instance of the VCE can be instantiated. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to an operating system feature where the kernel enables the existence of multiple isolated user-space instances called containers. These isolated user-space instances typically behave like actual computers from the perspective of the programs running within them. Computer programs running on a normal operating system can utilize all of that computer's resources, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, programs running inside a container can only use the contents of the container and the devices allocated to the container; this feature is known as containerization.

[0042] The private cloud 106 is similar to the public cloud 105, but differs in that its computing resources are available only for use by a single enterprise. While the private cloud 106 is illustrated as interacting with the WAN 102, in other embodiments, the private cloud may be completely isolated from the internet and accessible only through a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types), each often implemented by a different vendor. Each of the multiple clouds remains a separate and isolated entity, but in larger hybrid cloud architectures, they are integrated by standardized or proprietary technologies, enabling orchestration, management, and / or data / application portability across the multiple constituent clouds. In this embodiment, both the public cloud 105 and the private cloud 106 are part of a larger hybrid cloud.

[0043] A block diagram of query processing code 200 according to one embodiment of the present invention is illustrated in Figure 2. Specifically, the query processing code 200 includes a large-scale language model (LLM) module 210, a parsing module 220, an asset selection module 230, a validation module 240, and a query processing module 250. The LLM module 210, the parsing module 220, the asset selection module 230, the validation module 240, and the query processing module 250 may include one or more modules or units for performing various functions of the embodiments of the present invention described herein. The LLM module 210, the parsing module 220, the asset selection module 230, the validation module 240, and the query processing module 250 may be implemented by any combination of any number of software and / or hardware modules or units, and may reside in the volatile memory 112 of the computer 101 for execution by a processor such as the processor set 110.

[0044] The LLM module 210 can utilize one or more trained machine learning models to perform generation tasks, including the generation of database queries such as SQL statements, according to embodiments presented herein. The LLM module 210 may include one or more trained machine learning models, for example, a large-scale language model with a large number of weights (e.g., 20 billion or more). In various embodiments, the LLM module 210 may include a local model, or the LLM module 210 may access a model hosted in a network-accessible location. For example, the LLM module 210 may function as an Application Programming Interface (API) for an LLM. The LLM used by the LLM module 210 may be a general-purpose model, or it may be fine-tuned for the generation of database queries (e.g., SQL statements). Fine-tuning of the LLM may include additional training with training data that includes examples of natural language user queries and corresponding SQL statements.

[0045] Generally, the LLM module 210 can accept natural language user queries as input and generate database queries (e.g., SQL statements) as output based on those user queries. The LLM module 210 may be given prompts that include a user query and a request to generate an SQL statement based on that user query. The SQL statement generated by the LLM module 210 may include one or more columns, filter columns, and / or categorical filters. Specifically, columns may indicate the names of the columns to be searched, while filter columns may include additional filters that are imposed on any data retrieved from those columns. For example, in the SQL statement "SELECT name, age FROM users WHERE age > 18", "name" and "age" are columns selected from the table "users", and "age > 18" is a filter column that applies a command to return rows only if the user's age is greater than 18. Category filters are also filter conditions in the WHERE clause, where data follows specific, limited categories, such as marital status (e.g., "married" or "single"). For example, the SQL statement "SELECT * FROM products WHERE category IN ('Electronics','Clothing')" includes category filters for "electronics" and "clothing," instructing the query to select rows from the "products" table only if the row's "category" column contains entries that are either "electronics" or "clothing."As another example, LLM module 210 may take a natural language user query as input that includes "Show the best-selling products in Toronto, provided the store's revenue is over $10,000," and can generate an SQL statement that includes columns "sales" and "products," a filter column "revenue," and a category filter "Toronto."

[0046] In some embodiments, the LLM module 210 may be given one or more examples of desired output in the prompt. For example, the LLM may be given two example outputs (e.g., a two-shot prompt). In some embodiments, the prompt given to the LLM may include a requirement that the SQL statement generated by the LLM conform to a particular SQL dialect (e.g., MySQL). In some embodiments, the prompt may include a requirement that the LLM use minimal subqueries, particularly with respect to the WHERE clause. This reduces the processing requirements for extracting columns and filter columns in parsing. In some embodiments, the prompt may include instructions to prevent the LLM from performing mathematical calculations in the generated SQL statement. This may result in the benefit of reduced hallucination. In some embodiments, the prompt may include instructions for the LLM to perform chain-of-thought analysis, which breaks down the process of generating the SQL statement into multiple steps. Based on these instructions, the LLM is instructed to process them individually.

[0047] The parsing module 220 may include a parsing package capable of converting database queries into abstract syntax trees (ASTs). Specifically, the parsing module 220 can convert SQL statements into ASTs, which are machine-readable formats that make database queries executable on the database. In this way, the parsing module 220 transforms and refines queries to enable faster execution by structuring them into logical steps that can be executed in the optimal order. By parsing queries, complex queries, including subqueries or nested logic, can also be simplified, resulting in reduced processing requirements. The ASTs generated by the parsing module 220 may be analyzed, and columns, filter columns, and / or category filters may be extracted, which can be provided to the asset selection module 230.

[0048] The asset selection module 230 may perform semantic search operations to rank and identify databases relevant to user queries. Each database may be local (e.g., stored on computer 101) or remote (e.g., a remote database 130 on remote server 104). These databases may reside on any number of storage devices located in various network-accessible locations. The asset selection module 230 may analyze one or more indexes containing relevant data for multiple databases. These indexes may be supplemented with data obtained by web crawlers that discover databases on the internet and retrieve relevant data. In some embodiments, an index containing column names for multiple databases is provided. Additionally or alternatively, an index containing unique category values ​​sampled from each column of multiple databases may be provided. Column indexes and / or category value indexes may be used by the asset selection module 230 to perform semantic searches of databases in combination with columns, filter columns, and / or category filters provided by the parsing module 220.

[0049] The asset selection module 230 may employ various semantic search techniques to generate a confidence score for each database, each confidence score indicating the likelihood that a particular database contains data capable of supporting the execution of database queries generated based on natural language user queries according to the embodiments described herein. In this way, the asset selection module 230 can identify which databases are relevant to the user query. Since the user query does not specify a database, the schema of the database ultimately selected is unknown; therefore, the asset selection module 230 performs a semantic search to identify database columns and / or category filters that best match the column and / or category filters obtained by processing the user query using LLM.

[0050] In some embodiments, the asset selection module 230 performs a semantic search using the extracted column and filter column names. The semantic search can compare the extracted column and / or filter column names with entries in the column index to calculate a match score for each column and / or filter column. This process may include exact matches, fuzzy matches (e.g., stemming, headword recognition), and homogeneous matches. Thus, an overall match score for each database can be calculated using the degree to which the column names of each database in the column index match the extracted column and / or filter column names. For example, a database with more matches and / or closer matches (e.g., exact matches rather than fuzzy matches) may have a score indicating that it is more relevant than other databases with fewer matches and / or closer matches.

[0051] In some embodiments, semantic search can be performed using an LLM. This LLM may correspond to the LLM used by LLM module 210, or to a different LLM. In some embodiments, the LLM includes a general-purpose generative model. The LLM may generate synonyms for each input word (e.g., column name, filter column name), and these synonyms can then be used to search the column index in the same way that exact matching and / or fuzzy matching are performed.

[0052] In some embodiments, a semantic search is performed on each category filter value. Using an index of category filters, a match score may be calculated for each extracted category filter to identify databases that have category filters that match the extracted category filter. This score can indicate the extent to which a database contains the relevant category filter, and the score value may depend on the number of matches, the closeness of the matches (e.g., exact match, fuzzy match, and / or synonym match), and their homogeneity.

[0053] In some embodiments, the results of various semantic searches are combined to determine the overall confidence score for each database. Specifically, the overall confidence score for each database may be obtained by combining subscores obtained from (i) column and / or filter column matching, (ii) synonym matching by LLM, and (iii) category filter searching. In various embodiments, the subscores may be averaged, summed, or otherwise combined using any statistical or mathematical method. In some embodiments, different weights are applied to each subscore. For example, the column and / or filter column matching subscore may receive a greater weight than the synonym matching subscore by LLM, and the synonym matching subscore may then receive a greater weight than the subscore from category filter searching. Thus, the various subscores can have varying levels of influence on the resulting overall confidence score.

[0054] In some embodiments, if multiple search results from the same database correspond to the same extracted column name, only the result with the highest score may be retained. For example, if the extracted column name "Products" corresponds to both "Products" and "ProductSize" in a database, only the mapping to "Products" will be used because it is an exact match and therefore assigned a higher score.

[0055] Once a confidence score is determined for each database, the asset selection module 230 can sort the databases to select one or more relevant databases to use for executing database queries. In some embodiments, databases are sorted by their overall confidence score, and the most relevant database is selected as indicated by its confidence score value. In other embodiments, other considerations are included, such as the number of associated columns, the number of associated category filters, and homogeneity. In one embodiment, databases are sorted by the number of columns found to match extracted column names. For example, if three column names are extracted but only two are found to match in a particular database, that database will be sorted lower than another database where all three extracted column names appear. These databases may then be sorted by the number of category filter matches, and then by a comparison of the number of exact column name matches and synonym matches. Next, the databases may be sorted by their confidence score. As a result, the asset selection module 230 generates a ranked list of databases, each sorted by its ability to answer user queries. Next, several top-ranked databases may be selected for additional processing (e.g., validation, query processing, etc.). In some embodiments, the top-ranked databases are selected, while in other embodiments, a certain percentage of top databases (e.g., the top 10%) or a fixed number of top databases (e.g., the top 5) are selected.

[0056] The validation module 240 may perform operations to evaluate selected databases to determine whether each database can support the database queries that are thought to be relevant to the user query. In some embodiments, the validation module 240 utilizes an LLM for the validation task. This may be the same LLM used in the LLM module 210, or a different LLM (e.g., a generic LLM). The validation module 240 may provide the LLM with a prompt containing the user query and the database schema of the database being validated. This schema can be in any format (e.g., JavaScript Object Notation (JSON)) and may include column names, column descriptions, and the data type of each column. The prompt may further include a request to compare the schema with the user query and evaluate whether the database contains all the columns necessary to answer the user query. Thus, only validated databases may be used to execute the database query, thereby avoiding the consumption of unnecessary computing resources.

[0057] Once a database is selected and validated, a database query can be generated using the user query and the schema of the queried database (which is now known because a specific database has been selected). The LLM module 210 may generate the database query and answer the user query (included in the prompt) using the LLM, which is prompted to generate a complete query against a given database schema. Specifically, the LLM may be prompted to generate an SQL statement, and further, it may be required to generate an SQL statement conforming to a specific SQL dialect (e.g., MySQL). In some embodiments, the prompt may include several examples of the desired output (e.g., a 10-shot prompt). Furthermore, the prompt may instruct the LLM to discard all specific schemas and queries used in the examples provided in the prompt, thereby improving generalizability and the accuracy of the output. In some embodiments, the LLM may be prompted to use minimal subqueries, thereby reducing the processing requirements for parsing the resulting SQL statement. The model may also be prompted to reduce hallucination when generating filters.

[0058] The query processing module 250 may execute a database query generated by the LLM using the user query in combination with the schema of the database being queried. First, the parsing module 220 may parse the SQL statement and generate an AST, which may then be executed by the query processing module 250 on the selected database. In some embodiments, columns, filter columns, categorical filters, modifiers, aggregate functions, and / or other entities may be extracted from the AST for use in translating the SQL statement into other query specifications, depending on the query language supported by the underlying database. Once the results of the database query are obtained, the query processing module 250 may provide these results as a response to the user query. In some embodiments, these results may be provided to the LLM, which may be prompted with the user query and these results, as well as a request to generate a response to the user query. This natural language response may then be provided to the user device as an answer to the user query.

[0059] Referring here to Figure 3, a flowchart is provided relating to a process 300 for analyzing a database and generating a database query, according to one embodiment of the present invention. First, an input statement 302 (i.e., a user query) is received. The input statement 302 is processed in operation 304 to perform SQL generation by an LLM 306 (it should be understood that each LLM illustrated in process 300 may be the same LLM or different LLMs). The LLM 306 may be prompted to generate an SQL statement based on the input statement; this prompt may include other instructions to improve processing and accuracy, such as instructions to use a specific SQL dialect, instructions to use minimal subqueries, and instructions to generate output similar to the example of desired output provided in the prompt.

[0060] Operation 308 extracts the SELECT and WHERE clause entities from the resulting SQL statement. This SQL statement may be parsed and converted to an AST before extraction. The extracted entities may include column names, filter columns, and / or categorical filters. These entities may be used to analyze search indexes that include references to multiple databases to which the corresponding entities have been added from each database. Operation 310 performs a column name lookup. This involves identifying databases via indexes that have columns with names that are an exact or fuzzy match of the extracted column names. A column lookup database 318 may be accessed. This database may contain indexes generated based on the database schema 320 of each indexed database.

[0061] In operation 312, column synonym expansion is performed using LLM 314. The LLM may be prompted to generate synonyms for the column names extracted in operation 308; these synonyms may be used to perform a column synonym lookup in operation 316. In the column synonym lookup, column synonyms can be used for searching, similar to how column names are used in operation 310. Specifically, the column lookup database 318 is accessed to use the column name index for matching.

[0062] In operation 322, a data value lookup is performed to identify the category filters corresponding to the entities extracted in operation 308. The data lookup database 324 may contain an index on the unique category filters of each indexed database. This index may be added by analyzing the database values ​​326 to identify unique entries in the column. In some embodiments, a vector embedding 328 may be obtained for each unique entry for comparison with the vector embedding of the extracted entities in order to identify similar category filters in a vector space model using a similarity metric such as cosine similarity.

[0063] Operation 330 performs multi-factor asset selection using the results of operations 310, 312, and 322. Specifically, the databases may be sorted based on the degree of matching of the column names, column filters, and / or category filters in each database with respect to the entities extracted in operation 308. The degree of matching between the extracted entities and the database entities may be calculated to provide a confidence score for each database. This score may be a value based on various matches identified within each database. Different types of matches and their degrees can each affect the confidence score, and may be weighted separately so that certain types of matches and / or their degrees have a greater impact on the resulting confidence score than others.

[0064] Once the databases are ranked, one or more databases may be selected for further processing. Specifically, one or more of the most relevant databases may be selected; in operation 332, the corresponding schema may be retrieved for each database.

[0065] Operation 334 validates the schema of the selected database to determine whether the database can support the execution of a database query based on the user query. This schema may be validated using LLM336 by providing the user query and schema to the LLM in a prompt requesting the LLM to determine whether the database contains data available to respond to the user query. Schema validation may be performed in parallel with the execution of the database query to reduce computation time requirements. Once a database is selected, the query is executed against the selected database. In addition, auxiliary LLM calls may be executed in parallel to verify whether the selected schema can respond to the user query and reduce the possibility of hallucination. However, if it is determined that the selected database does not support the user query, the next highest-ranked database may be selected for validation and processing based on the database ranking.

[0066] In operation 338, LLM340 is used to generate an SQL statement using the schema of the selected database. LLM340 may be prompted with user query and schema details, along with other instructions, such as instructions to generate an SQL statement in a specific SQL dialect, instructions to generate an SQL statement using minimal subqueries, and similar instructions. In some embodiments, LLM340 is provided with short-shot prompts containing one or more examples of the desired output. In some embodiments, a timestamp corresponding to the timer of the user query is also provided to LLM to ensure that the database query retrieves data relevant at that moment.

[0067] Operation 342 generates a final output specification. This specification may include an executable AST generated by parsing the output of LLM340. The output specification is then executed on a selected, validated database to generate a response to a user query, which may then be provided to the user. In some embodiments, the results of query execution may be provided to the LLM to generate a natural language response to the user query.

[0068] Figure 4 is a flowchart relating to a method 400 for executing a user query according to one embodiment of the present invention.

[0069] In operation 410, an SQL statement is generated based on a user query provided by the user. The user query may include a natural language question, which is provided to the LLM by a prompt that also requests the generation of an SQL statement based on the query. This prompt may include other details, such as an example of the desired output (i.e., a one-shot or multi-shot prompt), a request to generate output with minimal subqueries, a request that the output conform to a specific SQL dialect, and similar details.

[0070] Operation 420 extracts entities from an SQL statement that include one or more columns, filter columns, and / or categorical filters. First, the SQL statement may be parsed to generate an AST from which the entities are extracted. However, in other embodiments, these entities may be extracted directly from the SQL statement. These entities may include one or more column names, filter columns, and categorical filters.

[0071] In operation 430, a semantic search of the database is performed using the extracted entities to determine the confidence score for each database. In some embodiments, column names and / or filter columns are evaluated against the column names in each database to identify exact matches and / or fuzzy matches. The number and / or proximity of these matches influence the confidence score. In some embodiments, LLM is used to generate synonyms for the column names and / or filter columns. These synonyms are then evaluated against the column names in each database as well. In some embodiments, a category filter is compared against the unique categories of the data entries in each database to determine the confidence score for each database. Various approaches to confidence score determination may be combined in any way to obtain the confidence score for each database.

[0072] Operation 440 selects databases based on their confidence scores. Databases may be ranked by their confidence scores, and one or more databases may be selected based on the confidence score indicating that the selected database is most likely to support the execution of SQL statements corresponding to user queries. In some embodiments, these databases may be evaluated using LLM to verify whether each selected database can support user queries.

[0073] Operation 450 generates an SQL statement using the user query and the schema of the selected database. Once the database is selected (and validated as necessary), its schema becomes known and may be provided to the LLM along with a request to generate an SQL statement specific to the user query and the database schema. As with the initially generated SQL statement, the LLM may be prompted with further instructions to refine the output.

[0074] Operation 460 validates the schema of the selected database. Specifically, the LLM may be provided with prompts that include a request to determine whether the selected database can support the execution of a user query, the schema of the selected database, and an SQL statement that will retrieve the relevant data in response to the user query. If the LLM returns a response indicating that the database can support such a query, that database may be used. However, if the LLM returns a response indicating that the database cannot support the query, another database (e.g., the next highest ranked database) may be selected for validation.

[0075] In operation 470, an SQL statement is executed on the selected database, and the query results are provided to the user. The SQL statement may be parsed to generate an AST that can be executed on the selected database to return the query results. In some embodiments, this query result may be provided to the LLM to generate natural language output as a response to the user query.

[0076] The embodiments described above and illustrated in the drawings should be understood as representing only a few of the many ways in which embodiments for automatically executing user queries using semantic searches to identify LLM and related databases can be implemented.

[0077] The environment of the embodiments of the present invention may include any number of computers or other processing systems (e.g., any computing device, treatment delivery device, computing service, etc.) and databases or other repositories organized in any desired format, and embodiments of the present invention may be applied to any desired type of computing environment (e.g., cloud computing, client-server, network computing, mainframe, standalone system, etc.). The computers or other processing systems used in embodiments of the present invention may be implemented with any number of personal or any other type of computer or processing system. These systems may include any type of monitor and input device (e.g., keyboard, mouse, speech recognition, etc.) for inputting and / or viewing information.

[0078] The software of embodiments of the present invention (e.g., LLM module 210, parsing module 220, asset selection module 230, verification module 240, query processing module 250, etc.) may be implemented in any desired computer language and should be understood to be developed by those skilled in the computer art based on the functional descriptions and flowcharts illustrated in the drawings contained herein. Furthermore, any reference herein to software performing various functions generally refers to a computer system or processor that performs these functions under the control of the software. The computer system of embodiments of the present invention may be implemented by any kind of hardware and / or other processing circuitry.

[0079] Various functions of a computer or other processing system may be distributed in any way across any number of software and / or hardware modules or units, processing systems or computer systems, and / or circuits. Computers or processing systems may be located locally or geographically separated from one another, and may communicate via any suitable communication medium (e.g., LAN, WAN, intranet, internet, hardwire, modem, wireless, etc.). For example, functions relating to embodiments of the present invention may be distributed in any way across various end-users / clients, distributed computing and server systems, and / or other intermediate processing devices. The software and / or algorithms described above and illustrated in the flowchart may be modified in any way to implement the functions described herein. Furthermore, the functions in the flowchart or description may be performed in any order to achieve the desired operation.

[0080] The communication network may be implemented by any number of communication networks of any kind (e.g., LAN, WAN, Internet, intranet, VPN, etc.). The computer or other processing system of the embodiment of the present invention may include conventional or any other communication devices for communicating over the network using conventional or any other protocol. The computer or other processing system may utilize any kind of connection (e.g., wired, wireless, etc.) to access the network. The local communication medium may be implemented by any suitable communication medium (e.g., local area network (LAN), wired connection, wireless link, intranet, etc.).

[0081] This system may use any number of conventional or other databases, datastores, or storage structures (e.g., files, databases, data structures, data, or other repositories) to store information. The database system may be implemented by any number of conventional or other databases, datastores, or storage structures (e.g., files, databases, data structures, data, or other repositories) to store information. The database system may be contained within or coupled to a computing system. The database system and / or storage structures may be separate from or local to a computer or other processing system. They may also store any desired data.

[0082] Embodiments of the present invention may use any number of user interfaces of any kind (e.g., a graphical user interface (GUI), command line, prompt, etc.) to retrieve or provide information (e.g., database queries, SQL statements, ASTs, column names, filter columns, category filters, database content, database schemas, indexes, etc.). This interface may contain any information organized in any format. The interface may contain any number of input or actuation mechanisms of any kind (e.g., buttons, icons, fields, boxes, links, etc.) placed anywhere to input / display information and to initiate desired actions via any appropriate input device (e.g., mouse, keyboard, etc.). The interface screen may contain any appropriate actuators (e.g., links, tabs, etc.) for navigating between screens in any format.

[0083] User query data, database query data, SQL statement data, AST data, column name data, filter column data, category filter data, database content, database schema data, and / or index data may include any information organized in any format and may be configured to provide users with desired information based on rules or other criteria.

[0084] Embodiments of the present invention are not limited to the specific tasks or algorithms described above, and may be used to answer any questions from a user, without requiring the user to specify a particular database to be used for answering such questions.

[0085] The data may include any format for storing the data, and may include any data describing the database, including metadata and its indexes. The data may be obtained by any means, accessed over a network, retrieved from local storage, provided by user input, and obtained by similar means.

[0086] The terms used herein are merely for the purpose of describing specific embodiments and are not intended to limit the invention. In this specification, singular nouns such as “a,” “an,” and “the” are intended to include plural nouns unless otherwise explicitly indicated in the context. It should be further understood that the terms “comprises,” “comprising,” “includes,” “including,” “has,” “have,” “having,” “with,” and similar terms, when used herein, indicate the existence of the features, items, steps, operations, elements, and / or components described herein, but do not preclude the existence or addition of one or more other features, items, steps, operations, elements, components, and / or sets thereof.

[0087] All corresponding structures, materials, actions, and equivalents of all means-plus-function elements or step-plus-function elements in the following claims are intended to include any structures, materials, or actions for carrying out the function in combination with any other claimed elements specifically claimed. The descriptions of various embodiments of the present invention are presented for illustrative purposes only and are not intended to be exhaustive or to limit the disclosed embodiments. Many changes and modifications will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best describe the principles of the embodiments, their practical applications or technical improvements to the art found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. The first step involves using a Large-Scale Language Model (LLM) to generate a first Structured Query Language (SQL) statement based on a user query provided by the user; A step of extracting multiple entities, including one or more columns, filter columns, and category filters, based on the first SQL statement; A step in which a semantic search is performed on the indexes of multiple databases using the one or more columns, filter columns, and category filters, and a confidence score is determined for each of the multiple databases, where the confidence score indicates the degree of match for each database with respect to the extracted one or more columns, filter columns, and category filters; The step of selecting a database based on the aforementioned confidence score and extracting the schema of the selected database; Using the aforementioned LLM, a second SQL statement is generated based on the user query and the schema; The step of executing the second SQL statement on the selected database and generating query results; and The stage of presenting the aforementioned query results to the user. A computer implementation method comprising the following:

2. The computer implementation method according to claim 1, further comprising the step of verifying the selected database using the LLM by providing the LLM with a prompt including the user query, the schema of the selected database, and a request for determining that the selected database contains data that supports the user query.

3. The computer implementation method according to claim 1 or 2, wherein the confidence score is determined by evaluating the column names of each database with respect to one or more extracted columns or filter columns.

4. The computer implementation method according to claim 1 or 2, wherein the confidence score is determined by generating synonyms for one or more extracted columns or filter columns and evaluating the column names of each database with respect to the generated synonyms.

5. The computer implementation method according to claim 1 or 2, wherein the confidence score is determined by evaluating one or more of the extracted category filters with respect to the category filters of each database.

6. The computer implementation method according to claim 1 or 2, wherein the prompts provided to the LLM for generating the first SQL statement or the second SQL statement include one or more examples of desired output.

7. The computer implementation method according to claim 1 or 2, wherein the prompts provided to the LLM for generating the first SQL statement or the second SQL statement include a request that the first SQL statement or the second SQL statement conform to a specific SQL dialect.

8. One or more memory locations; and At least one processor coupled to one or more of the aforementioned memories, wherein the at least one processor is Using a Large-Scale Language Model (LLM), a first Structured Query Language (SQL) statement is generated based on a user query provided by the user; Based on the first SQL statement described above, extract multiple entities that include one or more columns, filter columns, and category filters; Using the aforementioned one or more columns, filter columns, and category filters, a semantic search is performed on the indexes of multiple databases to determine the confidence score for each of the multiple databases, where the confidence score indicates the degree of match for each database with respect to the extracted one or more columns, filter columns, and category filters; A database is selected based on the aforementioned confidence score, and the schema of the selected database is extracted; Using the aforementioned LLM, a second SQL statement is generated based on the user query and the schema; Execute the second SQL statement on the selected database and generate query results; and The query results are presented to the user. It is structured in such a way. A computer system equipped with the following features.

9. The computer system according to claim 8, wherein the at least one processor is further configured to validate the selected database using the LLM by providing the LLM with prompts including a user query, the schema of the selected database, and a request for determining that the selected database contains data that supports the user query.

10. The computer system according to claim 8 or 9, wherein the confidence score is determined by evaluating the column names of each database with respect to one or more extracted columns or filter columns.

11. The computer system according to claim 8 or 9, wherein the confidence score is determined by generating synonyms for one or more extracted columns or filter columns and evaluating the column names of each database with respect to the generated synonyms.

12. The computer system according to claim 8 or 9, wherein the confidence score is determined by evaluating one or more of the extracted category filters with respect to the category filters of each database.

13. The computer system according to claim 8 or 9, wherein the prompts provided to the LLM for generating the first SQL statement or the second SQL statement include one or more examples of desired output.

14. The computer system according to claim 8 or 9, wherein the prompts provided to the LLM for generating the first SQL statement or the second SQL statement include a request that the first SQL statement or the second SQL statement conform to a specific SQL dialect.

15. At least one processor, A procedure for generating a first Structured Query Language (SQL) statement based on a user query provided by the user, using a Large-Scale Language Model (LLM); A procedure for extracting multiple entities, including one or more columns, filter columns, and category filters, based on the first SQL statement described above; A procedure for performing a semantic search on the indexes of multiple databases using the aforementioned one or more columns, filter columns, and category filters, and determining the confidence score for each of the multiple databases, wherein the confidence score indicates the degree of match for each database with respect to the extracted one or more columns, filter columns, and category filters; A procedure for selecting a database based on the aforementioned confidence score and extracting the schema of the selected database; A procedure for generating a second SQL statement based on the user query and the schema using the aforementioned LLM; A procedure for executing the second SQL statement on the selected database and generating query results; and A procedure for presenting the aforementioned query results to the user, A computer program designed to execute something.

16. The computer program according to claim 15, which causes the LLM to perform a procedure to verify the selected database by providing the LLM with a prompt including the user query, the schema of the selected database, and a request for determining that the selected database contains data that supports the user query.

17. The computer program according to claim 15 or 16, wherein the confidence score is determined by evaluating the column names of each database with respect to one or more extracted columns or filter columns.

18. The computer program according to claim 15 or 16, wherein the confidence score is determined by generating synonyms for one or more extracted columns or filter columns and evaluating the column names of each database with respect to the generated synonyms.

19. The computer program according to claim 15 or 16, wherein the confidence score is determined by evaluating one or more of the extracted category filters with respect to the category filters of each database.

20. The computer program according to claim 15 or 16, wherein the prompts provided to the LLM for generating the first SQL statement or the second SQL statement include one or more examples relating to the desired output.

Images