Apparatus, methods, and computer programs (validation of structured query language statements based on machine learning)

Abstract

Verify the accuracy of SQL commands. [Solution] An exemplary operation may include one or more of the following steps: receiving natural language input; running a generative machine learning (ML) model on the natural language input to generate a structured query language (SQL) statement; running the SQL statement on false simulated data to generate a first query result for the false simulated data; running the generative ML model on the natural language input and the false simulated data to generate a second query result for the false simulated data; and determining whether the SQL statement is valid based on a comparison of the first query result and the second query result.

Description

Technical Field

[0001] One of the most common mechanisms for accessing large amounts of structured data, such as tabular data, is through structured query language (SQL) commands. In recent years, machine learning has been used to generate SQL commands that are executable using generative capabilities, and then, in response to that, this SQL command is adopted to obtain data from a database. However, the SQL commands generated by a machine learning model are not completely accurate, and in many cases, manual work is required to verify the accuracy of the SQL commands. In addition, it is difficult for a user who is not an expert to distinguish whether the generated SQL command is accurate.

Summary of the Invention

Problems to be Solved by the Invention

[0002] The SQL commands generated by a machine learning model are not completely accurate, and in many cases, manual work is required to verify the accuracy of the SQL commands. In addition, it is difficult for a user who is not an expert to distinguish whether the generated SQL command is accurate. [[ID=十六]]

Means for Solving the Problems

[0003] One exemplary embodiment provides a device comprising memory and at least one processor communicatively coupled to the memory, wherein the at least one processor can perform one or more of the following: receiving natural language input; running a generative machine learning (ML) model on the natural language input to generate structured query language (SQL) statements; running the SQL statements on false simulated data to generate a first query result for the false simulated data; running the generative ML model on the natural language input and the false simulated data to generate a second query result for the false simulated data; and determining whether the SQL statements are valid based on a comparison of the first query result and the second query result.

[0004] Another exemplary embodiment provides a method which may comprise one or more of the following steps: receiving natural language input; running a generative machine learning (ML) model on the natural language input to generate a structured query language (SQL) statement; running the SQL statement on false simulated data to generate a first query result for the false simulated data; running the generative ML model on the natural language input and the false simulated data to generate a second query result for the false simulated data; and determining whether the SQL statement is valid based on a comparison of the first query result and the second query result.

[0005] A further exemplary embodiment provides a computer-readable storage medium comprising instructions, the instructions, when executed by a processor, causing the processor to perform one or more of the following steps: receiving natural language input; running a generative machine learning (ML) model on the natural language input to generate a structured query language (SQL) statement; running the SQL statement on false simulated data to generate a first query result for the false simulated data; running the generative ML model on the natural language input and the false simulated data to generate a second query result for the false simulated data; and determining whether the SQL statement is valid based on a comparison of the first query result and the second query result. [Brief explanation of the drawing]

[0006] [Figure 1] This figure illustrates a computing environment according to one embodiment of this solution.

[0007] [Figure 2] This diagram illustrates the process of generating SQL query statements and evaluating their accuracy based on examples and features of this solution.

[0008] [Figure 3A] This diagram illustrates the process of generating SQL commands using a generative ML model, based on examples and features of this solution.

[0009] [Figure 3B] This diagram illustrates the process of generating false simulated data based on examples and features of this solution.

[0010] [Figure 3C] This diagram illustrates the process of generating a first query result by querying fake simulated data based on SQL commands, using examples and features of this solution.

[0011] [Figure 3D] This diagram illustrates the process of generating a second query result by querying fake simulated data using a generative ML model, based on the examples and features of this solution.

[0012] [Figure 3E] This diagram illustrates the process of validating SQL commands based on the results of the first and second queries, using examples and features of this solution.

[0013] [Figure 3F] This diagram illustrates the process of executing validated SQL commands against productive data, based on examples and features of this solution.

[0014] [Figure 4] Figures 4A, 4B, and 4C illustrate different examples of verification results based on the characteristics of this solution.

[0015] [Figure 5A] This diagram illustrates a flowchart according to an exemplary embodiment.

[0016] [Figure 5B] This diagram illustrates a flowchart according to an exemplary embodiment. [Modes for carrying out the invention]

[0017] While this disclosure includes a detailed description of cloud computing, it should be understood that implementations of the teachings enumerated herein are not limited to cloud computing environments. Rather, embodiments of the solutions may be implemented in conjunction with any other type of computing environment that is currently known or may be developed in the future.

[0018] The exemplary embodiments are directed to an evaluation system that automatically validates the accuracy of SQL statements generated using a machine learning model, such as a large language model (LLM) having a generation ability. This system can evaluate the effectiveness of SQL generation in real-time, online, and automatically. This system can include a software application having a plurality of different modules that can perform different steps of the evaluation process.

[0019] For example, a first module of the software application can execute a generative machine learning (ML) model to generate an SQL statement. Here, the first module can receive a natural language input from a user device. The natural language input can include a human-readable description typed into a user interface or spoken into a microphone, and an identification of the data of interest. The generative ML model can receive the natural language input and the table schema of a database, and generate an SQL statement (e.g., a query, etc.) for retrieving the data of interest from the database.

[0020] A second module of the software application can generate fake synthetic data to test the accuracy of the SQL statement. As is understood, live data / valid data is subject to privacy requirements, regulations, confidentiality, and the like, and its use needs to be restricted. Therefore, such live data / valid data cannot be used for machine learning training, testing, or the like. To overcome these issues regarding live data / valid data, the exemplary embodiments generate fake synthetic data (which is similar to live data / valid data but is not subject to privacy requirements, regulations, confidentiality, and the like). Thus, the fake synthetic data can be used to test the accuracy of the SQL statement. The fake synthetic data can be generated based on the execution of the generative ML model by the second module. The execution can include the schema of the database.

[0021] The third module of the software application can query fake simulated data using the generated SQL statement to generate a first query result. The first query result may include tabular data extracted from the fake simulated data corresponding to the SQL query. Using the generated ML model, a fourth software application module can query the fake simulated data. For example, the fourth module can input natural language input and fake simulated data into the generated ML model (e.g., via a prompt), and execute the generated ML model against the prompt to generate a second query result.

[0022] The system can compare the first and second query results to determine whether the SQL statement is accurate. For example, the system can assume that the second query result generated by the generated ML model is accurate, and by comparing the first query result with the second query result, determine whether the first query result is accurate. If the first query result includes all the data of the second query result, the system can determine that the SQL statement is exactly accurate. In this case, the verified SQL statement can be executed against live data / valid data to execute the original query requested by the natural language input. However, if the first query result does not include all the data from the second query result, the system can determine that the SQL statement is inaccurate and may not execute the SQL statement against valid data. Instead, the system can send an error notification to the graphical user interface (GUI) of the software application, thereby requesting the user to input different natural language inputs.

[0023] According to various embodiments, SQL statements (or commands) can be automatically generated in response to user queries, and the accuracy of query results can be compared using simulated data. Automatic, unsupervised correction of SQL statements can be performed, saving human and material resources and helping users and / or systems verify the accuracy of results.

[0024] In an exemplary embodiment, a generative machine learning model (e.g., LLM) is more suitable for obtaining answers from documents. However, private documents have confidentiality constraints and cannot be used with publicly available pre-trained machine learning models. Furthermore, these documents and associated data are often large in scale, and if directly fed into a generative machine learning model, it can cause token overflow problems. In this context, the system can simulate false data according to the database table schema information, and by controlling the size of the simulated false data, it can overcome the constraints regarding the confidentiality of tokens and datasets.

[0025] The system described herein for generating and evaluating the accuracy of SQL statements may be implemented within a software application, service, or similar, which may be hosted by a hosting platform such as a cloud platform, web server, database, or similar.

[0026] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and deployed with minimal management effort or interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0027] The characteristics are as follows:

[0028] On-demand self-service: Cloud consumers can unilaterally provision computing power, such as server time and network storage, automatically as needed, without requiring human interaction with the service provider.

[0029] Broad network access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use with heterogeneous thin-client or thick-client platforms (e.g., mobile phones, laptops, and PDAs®).

[0030] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated as needed. Consumers generally do not have control or knowledge of the exact location of the resources provided, but they may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center), thus demonstrating location independence.

[0031] Rapid resilience: Capabilities are provisioned quickly and flexibly, sometimes automatically, allowing for rapid scaling out or rapid release and rapid scaling in. To consumers, the available capacity for provisioning often appears unlimited and can be purchased in any quantity at any time.

[0032] Measurement Services: Cloud systems automatically control and optimize resource usage by leveraging measurement capabilities appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts) at a certain level of abstraction. Resource usage can be monitored, controlled, and reported, providing transparency to service providers and consumers.

[0033] The service model is as follows:

[0034] Software as a Service (SaaS): Consumers can use a provider's applications on a cloud infrastructure. These applications are accessible from various client devices through thin client interfaces such as web browsers (e.g., web-based email). Except for limited user-specific application configuration settings, consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or individual application capabilities.

[0035] Platform as a Service (PaaS): A provider provides consumers with the ability to deploy applications they have created or acquired onto a cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and, if applicable, the configuration of the application hosting environment.

[0036] Infrastructure as a Service (IaaS): The ability provided to consumers is to provision processing, storage, networking, and other basic computing resources, allowing consumers to deploy and run any software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they have control over the operating system, storage, and deployed applications, and, in some cases, limited control over selected networking components (e.g., host firewalls).

[0037] The deployment model is as follows:

[0038] Private Cloud: Cloud infrastructure is operated for a single organization only. That organization or a third party may manage it, and it may reside on-premises or off-premises.

[0039] Community Cloud: Cloud infrastructure is shared by multiple organizations to support a specific community with common interests (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can reside on-premises or off-premises.

[0040] Public cloud: Cloud infrastructure is made available to the general public or large industry groups and is owned by an organization that sells cloud services.

[0041] Hybrid Cloud: This cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain separate entities but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing across clouds).

[0042] Service-oriented cloud computing environments emphasize statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing lies an infrastructure that includes a network of interconnected nodes.

[0043] The features, structures, or characteristics described herein may be combined or omitted in any preferred manner in one or more embodiments. For example, the phrases “exemplary embodiments,” “some embodiments,” or other similar wording used throughout this specification refer to the fact that certain features, structures, or characteristics described in relation to such embodiments may be included in at least one embodiment. Thus, the phrases “exemplary embodiments,” “some embodiments,” “other embodiments,” or other similar wording appearing throughout this specification do not necessarily all refer to the same group of embodiments, and the features, structures, or characteristics described may be combined or omitted in any preferred manner in one or more embodiments. Furthermore, even if the connections shown in the figures are unidirectional or bidirectional arrows, any connection between elements may allow unidirectional and / or bidirectional communication. Also, any device shown in the figures may be a different device. For example, if a mobile device is shown transmitting information, a wired device may also be used to transmit information.

[0044] Figure 1 illustrates a computing environment 100 according to one embodiment of this solution. Various aspects of this disclosure are described by explanatory text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of computer program products (CPPs). With respect to any flowchart, depending on the technology involved, operations may be performed in a different order than those shown in a given flowchart. For example, also depending on the technology involved, two operations shown in consecutive blocks of a flowchart may be performed in reverse order, as a single integrated step, simultaneously, or with at least partial time overlap.

[0045] Embodiments of a computer program product ("CPP Embodiment" or "CPP") are terms used in this disclosure to describe any set of one or more storage media (also called "Multiple Media") that collectively comprise a set of one or more storage devices and collectively comprise machine-readable code corresponding to instructions and / or data for performing computer operations specified in a given CPP claim. "Storage Device" is any tangible device capable of holding and storing instructions by a computer processor. Computer-readable storage media may, but are not limited to, electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, mechanical storage media, or any preferred combination thereof. Some known types of storage devices, including these media, include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices (such as pits / lands formed on the main surface of a punch card or disk), or any suitable combination of the foregoing. Computer-readable storage media cannot be interpreted as storage in the form of temporary signals when the term is used in this disclosure. As will be understood by those skilled in the art, data is typically moved at several intermittent points in the normal operation of the storage device, such as during access, defragmentation, or garbage collection, but this does not mean that the storage device is temporary, as data is not temporary while it is stored.

[0046] Referring to Figure 1, the computing environment 100 includes an example of an environment for executing at least a portion of the computer code involved in performing the method of the present invention, such as an SQL generation and validation system 116. In addition to block 116, the computing environment 100 includes, for example, a computer 101, a wide area network (WAN 102), an end-user device (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, the computer 101 includes a processor set 110 (including processing circuits 120 and a cache 121), a communication fabric 111, volatile memory 112, persistent storage 113 (including an operating system 122 and the block 116 identified above), a peripheral device set 114 (including a user interface (UI) device set 123, storage 124, and an Internet of Things (IoT) sensor set 125), and a network module 115. The remote server 104 includes a remote database 130. The public cloud 105 includes a gateway 140, a cloud orchestration module 141, a host physical machine set 142, a virtual machine set 143, and a container set 144.

[0047] Computer 101 can take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch, or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device, currently known or to be developed in the future, capable of running programs, accessing networks, or querying databases such as remote database 130. As is well understood in the field of computer technology, and depending on the technology, the execution of a computer implementation can be distributed among multiple computers and / or multiple locations. On the other hand, in this presentation of computing environment 100, in order to keep the presentation as concise as possible, the detailed discussion focuses on a single computer, specifically computer 101. Although computer 101 is not shown in the cloud in Figure 1, it can be located in the cloud. On the other hand, computer 101 does not need to be located in the cloud, except to any extent that can be definitively shown.

[0048] The processor set 110 includes one or more computer processors of any type currently known or to be developed in the future. The processing circuitry 120 may be distributed across multiple packages, for example, multiple coordinated integrated circuit chips. The processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. The cache 121 is memory located within the processor chip package and is typically used for data or code that should be available for high-speed access by threads or cores running on the processor set 110. The cache memory is typically organized into multiple levels depending on its relative proximity to the processing circuitry. Alternatively, some or all of the cache for the processor set may be located "off-chip". In some computing environments, the processor set 110 may operate with qubits and be designed to perform quantum computing.

[0049] Computer-readable program instructions are typically loaded onto computer 101, causing the processor set 110 of computer 101 to execute a series of operational steps, thereby influencing a computer implementation method. As a result, the instructions thus executed instantiate the methods specified in the flowcharts and / or descriptions of the computer implementation methods contained herein (collectively referred to as the "Methods of the Invention"). These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 121 and other storage media discussed below. The program instructions and associated data are accessed by the processor set 110 to control and direct the execution of the Methods of the Invention. In computing environment 100, at least some of the instructions for executing the Methods of the Invention may be stored in block 116 in persistent storage 113.

[0050] The communication fabric 111 is a signal conduction path that enables various components of the computer 101 to communicate with one another. Typically, this fabric comprises switches and conductive paths such as buses, bridges, physical input / output ports, and similar components. Other signal communication paths, such as optical fibers and / or wireless, may be used.

[0051] Volatile memory 112 is any type of volatile memory currently known or to be developed in the future. Examples include dynamic random access memory (RAM) or static RAM. Volatile memory typically features random access, but this is not mandatory unless explicitly stated. In computer 101, volatile memory 112 is located in a single package and resides inside computer 101, but alternatively or additionally, volatile memory may be distributed across multiple packages and / or located externally with respect to computer 101.

[0052] The persistent storage 113 is any form of non-volatile computer storage that is currently known or may be developed in the future. The non-volatility of this storage means that the stored data is maintained regardless of whether power is supplied to the computer 101 and / or directly to the persistent storage 113. The persistent storage 113 may be read-only memory (ROM), but typically at least a portion of the persistent storage allows for writing, deleting, and rewriting of data. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. The operating system 122 can take multiple forms, such as various known proprietary operating systems or open-source portable operating system interface type operating systems that employ a kernel. The code contained in block 116 typically includes at least a portion of computer code involved in performing the method of the present invention.

[0053] The peripheral device set 114 includes a set of peripheral devices for the computer 101. Data communication connections between the computer 101's peripheral devices and other components can be implemented in various ways, including Bluetooth® connections, Near-Field Communication (NFC) connections, connections made by cables (such as Universal Serial Bus (USB) type cables), insert-type connections (e.g., Secure Digital (SD) cards), connections made through local area communication networks, and even connections made through wide area networks such as the Internet. In various embodiments, the UI device set 123 may include components such as a display screen, speaker, microphone, wearable devices (such as goggles and smartwatches), keyboard, mouse, printer, touchpad, game controller, and haptic devices. Storage 124 is external storage such as an external hard drive, or insertable storage such as an SD card. Storage 124 may be persistent and / or volatile. In some embodiments, storage 124 may take the form of a quantum computing memory device for storing data in the form of qubits. In embodiments where computer 101 is required to have a large amount of storage (for example, when computer 101 locally stores and manages a large database), this storage may be provided by peripheral storage devices designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor set 125 comprises sensors that can be used in Internet of Things applications. For example, one sensor may be a thermometer and another may be a motion detector.

[0054] The network module 115 aggregates computer software, hardware, and firmware that enable computer 101 to communicate with other computers via the WAN 102. The network module 115 may include hardware such as a modem or Wi-Fi® signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or web browser software for communicating data over the Internet. In some embodiments, the network control and network forwarding functions of the network module 115 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing Software-Defined Networking (SDN)), the control and forwarding functions of the network module 115 are performed on physically separate devices, such that the control function manages multiple different network hardware devices. Computer-readable program instructions for performing the method of the present invention can typically be downloaded to computer 101 from an external computer or external storage device via a network adapter card or network interface included in the network module 115.

[0055] WAN102 is any wide area network (e.g., the Internet) capable of communicating computer data over non-local distances by any currently known or future-developed technologies. In some embodiments, the WAN may be replaced and / or complemented by a local area network (LAN), such as a Wi-Fi® network, designed to communicate data between devices within a local area. WANs and / or LANs typically include computer hardware such as copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0056] The end-user device (EUD) 103 is any computer system used and controlled by an end-user (e.g., a customer of the company operating computer 101) and can take any of the forms discussed above in relation to computer 101. The EUD 103 typically receives useful and valuable data from the operation of computer 101. For example, in a hypothetical scenario where computer 101 is designed to provide recommendations to an end-user, these recommendations would typically be communicated from the computer 101's network module 115 to the EUD 103 via the WAN 102. In this way, the EUD 103 can display, or otherwise present, the recommendations to the end-user. In some embodiments, the EUD 103 may be a client device such as a thin client, heavy client, mainframe computer, desktop computer, etc.

[0057] The remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. The remote server 104 may be controlled and used by the same entity that operates computer 101. The remote server 104 represents a machine that collects and stores data that is beneficial and useful to other computers, such as computer 101. For example, in a hypothetical scenario where computer 101 is designed and programmed to provide recommendations based on historical data, this data may be provided to computer 101 from the remote database 130 of the remote server 104.

[0058] The public cloud 105 is any computer system available for use by multiple entities, providing on-demand availability of computer system resources and / or other computing capabilities, particularly data storage (cloud storage) and computing power, without direct active management by the user. Cloud computing typically leverages resource sharing to achieve coherence and economies of scale. Direct active management of the public cloud 105 computing resources is performed by the computer hardware and / or software of the cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by virtual computing environments running on various computers that make up the host physical machine set 142, which is the universe of physical computers within and / or available to the public cloud 105. The virtual computing environment (VCE) typically takes the form of virtual machines from the virtual machine set 143 and / or containers from the container set 144. These VCEs can be stored as images and transferred between and between various physical machine hosts, either as images or after VCE instantiation. The cloud orchestration module 141 manages the transfer and storage of images, deploys new VCE instantiations, and manages active instantiations of VCE deployments. The gateway 140 aggregates computer software, hardware, and firmware, enabling the public cloud 105 to communicate over the WAN 102.

[0059] Here, some further explanation of virtualized computing environments (VCEs) is provided. A VCE can be stored as an "image." A new active instance of a VCE can be instantiated from an image. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to an operating system feature where the kernel allows for the existence of multiple isolated user-space instances called containers. These isolated user-space instances typically behave like a real computer in terms of the programs running within them. A computer program running on a normal operating system can utilize all of that computer's resources, including connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, a program running inside a container can only use the container's contents and the devices allocated to the container; this feature is known as containerization.

[0060] The private cloud 106 is similar to the public cloud 105, except that the computing resources are available only for a single enterprise. Although the private cloud 106 is shown as being in communication with the WAN 102, in other embodiments the private cloud may be completely isolated from the internet and accessible only through a local / private network. A hybrid cloud consists of multiple clouds of different types (e.g., private cloud, community cloud, or public cloud type), often implemented by different vendors. Each of the multiple clouds remains a separate, discrete entity, but the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the multiple configured clouds. In this embodiment, both the public cloud 105 and the private cloud 106 are part of a larger hybrid cloud.

[0061] Figure 2 illustrates a process 200 that generates and evaluates the accuracy of SQL statements, based on an example and features of this solution. For example, the process 200 shown and described in Figure 2 may be executed by a software application using multiple code modules capable of executing each step. Referring to Figure 2, in step 202, the process may include a procedure to run an ML model on natural language input to generate SQL statements. The natural language input may include human-typed or human-spoken descriptions of the data that the user wishes to extract / display.

[0062] In step 204, the process may include a procedure for generating false simulated data for use when testing the accuracy of SQL commands. As described herein, valid data may be subject to privacy requirements, regulations, confidentiality, and similar, and therefore its use may be restricted. To overcome these challenges with valid data, exemplary embodiments generate false simulated data that is similar to actual valid data but is not subject to privacy requirements, regulations, confidentiality, etc. Here, the false simulated data may be generated by inputting prompts into the ML model that include the table schemas of the databases for the valid data and the false simulated data. In some embodiments, the prompts may also include restrictions specifying the number of rows, columns, etc., of data that will be included in the false simulated data. This allows the system to limit the size of the false simulated data and prevent issues such as token overflow problems.

[0063] In step 206, the system may query fake simulated data using the generated SQL commands to produce a first query result. The first query result may include tabular data (e.g., a subset of data) extracted from the fake simulated data based on the statements in the SQL commands. In step 208, the system may query the fake simulated data using the generated ML model. For example, the system may input a prompt containing natural language input and fake simulated data into the ML model to produce a second query result. It should be understood that steps 206 and 208 may be executed simultaneously. Alternatively, steps 206 and 208 may be executed sequentially, with either step 206 or 208 being executed first.

[0064] In step 210, the system may verify the accuracy of the SQL command generated by the ML model in step 202. The system may determine whether the SQL command is accurate by comparing the first and second query results. For example, the system may assume that the second query result generated by the ML model is accurate and determine whether the SQL command is accurate by comparing the first query result (the result of executing the SQL command) with the second query result. If the first query result includes all the data from the second query result, the system may determine that the SQL statement is sufficiently accurate. In this case, the validated SQL statement can be executed on live / valid data to execute the original query requested by the natural language input. However, if the first query result does not include all the data from the second query result, the system may determine that the SQL statement is inaccurate and does not need to be executed on valid data. Instead, the system may send an error notification to the graphical user interface of the software application, thereby requesting the user (or system) to enter different natural language input.

[0065] Figure 3A illustrates a process 300A that generates an SQL command 330 using ML model 322, based on an example and features of this solution. Referring to Figure 3A, a host system, such as a software application 320, may host ML model 322. In this example, ML model 322 is pre-trained and may be capable of performing multiple tasks, including generating SQL commands, simulated data, query results, etc. For example, ML model 322 may be improved (or fine-tuned) through an additional training process that includes running ML model 322 on additional examples of SQL command and natural language input pairs, SQL commands and query results, and similar.

[0066] In one example, a user may input natural language input 316 into a software application 320 through a computing system 310. The software application 320 may be hosted on a host platform (not shown) to which the computing system 310 can connect via a network such as the Internet. The software application 320 may also include a progressive web application (PWA) accessible through a web browser installed on the computing system 310 by entering the URL of the PWA into the web browser. The computing system 310 may include a mobile device that can download and access the software application 320 from a marketplace or similar.

[0067] The computing system 310 includes a display device 312 that allows a user to input content, such as text, which can be used to generate natural language input 316, and submit the natural language input 316 to a software application 320. The computing system 310 may include a microphone 314 that can capture voice input from the user and generate natural language input 316. The computing system 310 may submit the natural language input 316 to the software application 320. As shown in Figure 3B, the natural language input may include a request for data from a database (such as database 340). In response, the software application 320 may generate an SQL command 330 containing a statement to retrieve the requested data from the database.

[0068] SQL command 330 may include a SELECT clause that identifies data variables / columns; a FROM clause that identifies one or more data tables from which data is retrieved; a GROUP BY clause used to group rows of data in a table based on matching values ​​in one or more columns; an ORDER BY clause that sorts the results in ascending or descending order within a specified column; and similar clauses.

[0069] Figure 3B illustrates a process 300B that generates false simulated data, based on an example and features of this solution. Referring to Figure 3B, a software application 320 may prompt an ML model 322 to input a prompt containing the data type of the data in an SQL command 330, a business description of the requested data, the database schema 342 of database 340, the data format, and similar information. In response, the ML model 322 may generate false simulated data 350 that resembles the data in database 340, including the database schema 342, format, and data type. The ML model 322 may take the prompt and create one or more tables of data (false simulated data 350) using false data that resembles one or more tables in database 340 but is not subject to privacy requirements, regulatory requirements, confidentiality requirements, and similar information.

[0070] The false simulated data 350 may contain columns, rows, and similar elements to those in database 340. In addition, the size of the false simulated data 350 may be limited so that it can be input into the ML model 322 and run efficiently.

[0071] Figure 3C illustrates a process 300C that generates a first query result 352 by querying false simulated data 350 based on an SQL command 330, according to an example and features of this solution. Referring to Figure 3C, the software application 320 may include a query engine 324 that can query false simulated data 350 based on an SQL command 330. For example, the query engine 324 may interpret the instructions in the SQL command 330 and, based on the instructions in the SQL command 330, extract a subset of data (e.g., tabular data) from the false simulated data 350 to generate a first query result 352. In this example, the query engine 324 takes in the SQL command 330 and extracts the first query result 352, which is held in the memory (not shown) of the software application 320.

[0072] Figure 3D illustrates a process 300D that uses ML model 322 to query false simulated data 350 and generate a second query result 354, based on an example and features of this solution. Referring to Figure 3D, instead of relying on SQL commands 330, ML model 322 can directly query the false simulated data 350 using natural language input 316 and the false simulated data 350. For example, a software application 320 may generate a prompt containing natural language input 316 and the false simulated data 350 and input this prompt to ML model 322. In response, ML model 322 may generate a second query result 354 from the false simulated data 350.

[0073] Figure 3E illustrates a process 300E for validating an SQL command based on the first and second query results, according to an example and features of the present solution. In an exemplary embodiment, the second query result 354 generated by the ML model 322 can be considered / assumed to be accurate. Therefore, the second query result 354 can be used to verify the accuracy of the first query result 352 generated by the execution of the SQL command 330. In this case, the software application 320 can compare the first query result 352 with the second query result to determine whether the SQL command 330 is accurate.

[0074] For example, if the first query result 352 contains all the content in the second query result 354, the first query result 352 may be considered correct, and similarly, the SQL command 330 may be considered correct. This includes situations where the first query result 352 contains more / additional data than what is contained in the second query result 354. The first query result 352 contains the content of the second query result 354; even if the first query result 352 has additional content not contained in the second query result 354, it is still considered correct. In the example in Figure 3E, the first query result 352 is the same as (or identical to) the second query result 354, and is therefore considered correct. Here, the software application 320 may determine that the result 360 of the SQL command 330 is valid.

[0075] Figure 3F illustrates a process 300F that executes a validated SQL command against valid data, based on an example and features of this solution. Referring to Figure 3F, if the software application 320 determines that the SQL command 330 is valid (or correct), the software application 320 may invoke the query engine 324 to execute the SQL command 330 against live data (or valid data) in the database 340. In response, the query engine 324 may generate a valid result 344 of the SQL command 330. The valid result 344 may be provided as output to another software application. As another example, the valid result 344 may be displayed on a GUI or similar.

[0076] Figures 4A to 4C illustrate examples of this solution and different validation results based on its features. For example, Figure 4A illustrates the first example 400 where the first query result (generated by executing an SQL command on fake simulated data) and the second query result (generated by running an ML model on fake simulated data) are equal. This example is considered a successful validation because all the content of the second query result is included in the first query result.

[0077] Figure 4B illustrates a second example, 410, where the first query result (generated by executing an SQL command on fake simulated data) and the second query result (generated by running an ML model on fake simulated data) are not equal, but the first query result contains all the content of the second query result. This example is also considered a successful validation because, although the first query result contains additional content, the entire content of the second query result is included in the first query result.

[0078] Figure 4C illustrates a third example, 420, where the first query result (generated by executing an SQL command on fake simulated data) and the second query result (generated by running an ML model on fake simulated data) are not equal, and the first query result does not contain all the content of the second query result. This example is considered a failure because the entire content of the second query result is not included in the first query result. In this example, the SQL command is considered incorrect and may not be used, and the software application may prompt the user for new natural language input.

[0079] The exemplary solutions presented utilize machine learning models such as neural networks, but other branches of AI, including but not limited to computer vision, fuzzy logic, expert systems, deep learning, generative AI, and natural language processing, may be employed in developing the machine learning models used in these solutions. Furthermore, the machine learning models included in these examples and features of the solutions are not limited to any specific AI algorithm. Any algorithm or combination of algorithms relating to supervised, unsupervised, and reinforcement learning may be employed.

[0080] The AI ​​models, ML models, neural networks, and other lineages of AI described and / or illustrated herein are built upon the fundamental principles of prior art and form the foundation for all future technological advancements in artificial intelligence. The AI ​​classification system describes the stages of progress and development of AI. The first classification is known as "reactive machines," followed by the modern AI classification of "limited memory machines" (also known as "artificial narrow intelligence"), then evolving to "theory of mind" (also known as "artificial general intelligence"), and finally reaching the AI ​​classification of "self-aware" (also known as "artificial superintelligence"). Modern limited memory machines are an expanded group of AI models built upon their predecessors, reactive machines. Reactive machines mimic human responses to stimuli; however, their capabilities are limited because they typically cannot learn from previous experiences. With the emergence of AI models' learning capabilities, their classification was elevated to that of limited-memory machines. In this modern classification, AI models inherit all the capabilities of reactive machines while learning from large amounts of data, detecting patterns, solving problems, generating and predicting data, and similar tasks.

[0081] Examples of AI models classified as limited memory machines include, but are not limited to, chatbots, virtual assistants, machine learning, neural networks, deep learning, natural language processing, generative AI models, and any future AI models yet to be developed that possess the characteristics of a limited memory machine.

[0082] For example, neural networks are machine learning models that learn associations and connections based on training data, improving their accuracy for fast data classification, clustering, and other data analysis. Such neural network capabilities form the foundation of current deep learning models and are the basic building blocks for models yet to be developed.

[0083] For example, generative AI models combine limited memory machine learning and deep learning with limited memory machine learning techniques to form the foundational building blocks for future AI models. For instance, theory of mind is the next advancement in AI, where AI models may be able to perceive, connect, and react by generating appropriate responses in response to the entities they interact with; all of these theory of mind capabilities rely on the fundamental principles of generative AI. Furthermore, in the evolution to self-aware classification, AI models will be able to understand and evoke emotions in the entities they interact with, and may even possess their own emotions, beliefs, and needs, all of which rely on the fundamental principles of generative AI, where they learn from experience and generate and derive conclusions about themselves and their surroundings.

[0084] An AI model may include, but is not limited to, at least one machine learning model, neural network model, deep learning model, generative AI model, or any combination of models from the lineage of AI. AI models are essential for future artificial intelligence models. As used herein, AI models refer to both modern and future AI models.

[0085] Artificial intelligence (AI) systems are built and trained to perform various tasks in an automated manner. For example, AI systems can receive and understand oral and / or written dialogues and function as digital assistants, speech-to-text programs, etc. Other AI systems are trained on different types of information, enabling trained systems to generate content such as new works of art based on observed styles or new compound ideas based on the history of chemical research.

[0086] The foundational model is an artificial intelligence system trained on a broad set of unlabeled data that can be used for different tasks with minimal fine-tuning. The unlabeled data, in some instances, includes images and / or language. In response to short prompts input to the foundational model, the system generates outputs, such as an entire essay or a complex image, based on the parameters outlined in the input prompt. The foundational model may generate outputs that attempt to satisfy the parameters even if it has never been trained with specific training data containing those exact parameters—for example, for its exact arguments or to generate images in that manner.

[0087] By using self-supervised and transfer learning, foundational models can apply the information they have learned about one situation to another. For example, a human can learn how to drive one car and, without excessive effort, learn how to drive other vehicles such as cars, trucks, or buses. Foundational models are similarly used to reach proficiency in several new areas without being trained entirely from scratch. Foundational models appear to possess inherent creativity in tasks such as piecing together a coherent argument or creating an original work of art. Foundational models are well-established in natural language processing techniques. One example of how useful foundational models are is that, with respect to previous generations of AI techniques, if one wanted to build an AI model that could summarize the body of text for itself, tens of thousands of labeled examples would be required just for the summarization use case. Using pre-trained foundational models dramatically reduces the labeling data requirement. Firstly, domain-specific foundational models are created by fine-tuning them with domain-specific unlabeled corpora. Next, a foundational model for summarization is trained using a much smaller amount of labeled data, potentially as few as a thousand labeled examples. This domain-specific foundational model can be used for many tasks, replacing traditional techniques that required building a model from scratch for each use case. The foundational model is even applicable to computer programming, code analysis, generation, and repair.

[0088] Several foundational models are being used for sentiment analysis. Using pre-trained foundational models, it's possible to train sentiment analysis for new languages ​​using only a few thousand sentences, reducing the number of annotations required to one-hundredth of that of traditional models. Reducing labeling requirements makes implementation in various technical domains far easier. Systems that perform specific tasks within a single domain are being replaced by broader AI that learns more comprehensively and functions across multiple domains and problems. Foundational models trained on large, unlabeled datasets and fine-tuned for various applications are driving this shift.

[0089] Large-scale language models (LLMs) are foundational models trained on massive amounts of data, enabling them to understand and generate natural language and other types of content to perform a variety of tasks. LLMs are implemented at different levels to enhance their natural language understanding (NLU) and natural language processing (NLP) capabilities. This development of LLMs has occurred in parallel with the development of machine learning, machine learning models, algorithms, neural networks, and transformer models that provide the architecture for these AI systems.

[0090] LLM is a foundational model trained on vast amounts of data to drive multiple use cases and applications and provide the foundational capabilities to solve many tasks. This LLM concept stands in stark contrast to the idea of ​​individually building and training domain-specific models for each use case, which is prohibitively expensive under many criteria (most importantly cost and infrastructure), hinders synergies, and can even lead to performance degradation.

[0091] LLM represents a significant breakthrough in NLP and artificial intelligence. LLM is accessible through interfaces such as OpenAI's Chat GPT-3 and GPT-4, which have received support from Microsoft®. Other examples include Meta's Llama model, Google's Transformer-based bidirectional encoder representations (BERT / RoBERTa), and the PaLM model. IBM has also recently launched its Granite model series on watsonx.ai, which serves as the backbone of generative AI for other IBM products such as watsonx Assistant and watsonx Orchestrate.

[0092] In short, LLMs are designed to understand and generate text and other forms of content as if they were human, based on the vast amount of data used to train them. They can infer from context, generate consistent and contextually relevant responses, translate into languages ​​other than English, summarize text, answer questions (general conversations and FAQs), and even assist with creative document creation or code generation tasks. LLMs can perform some or all of these tasks thanks to a large number of parameters, such as billions, that enable them to capture complex patterns in language and perform a wide variety of language-related tasks. LLMs are transforming applications in various fields, from chatbots and virtual assistants to content generation, research support, and language translation.

[0093] LLMs operate by leveraging deep learning techniques and vast amounts of text data. These models are typically based on transformer architectures, such as generative pre-trained transformers, which excel at handling sequential data like text input. An LLM consists of multiple layers of neural networks, each with parameters that can be fine-tuned during training, and these are further enhanced by layers known as attention mechanisms that focus on specific parts of the dataset.

[0094] Figure 5A illustrates a flowchart of Method 500 according to an exemplary embodiment. Referring to Figure 5A, in 501, the Method may include a step of receiving natural language input. In 502, the Method may include a step of running a generative machine learning (ML) model on the natural language input to generate a structured query language (SQL) statement. In 503, the Method may include a step of running the SQL statement on false simulated data to generate a first query result for the false simulated data. In 504, the Method may include a step of running a generative ML model on the natural language input and the false simulated data to generate a second query result for the false simulated data. In 505, the Method may include a step of determining whether the SQL statement is valid based on a comparison of the first query result and the second query result.

[0095] Figure 5B illustrates a flowchart of Method 510 according to an exemplary embodiment. Referring to Figure 5B, in 511, the Method may have a step of determining that an SQL statement is valid if the content included in the first query result includes all the content included in the second query result. In 512, the Method may have a step of determining that an SQL statement is invalid if the content included in the first query result does not include all the content included in the second query result. In 513, in response to the determination that the SQL statement is valid, the Method may have a step of executing the SQL statement against valid data stored in the database to generate a valid query result, and a step of outputting the valid query result to a software application.

[0096] In 514, the false simulated data may include tabular data, and the method may include a step of generating false simulated data by running a generative ML model on the database schema and the data types associated with the SQL statement before executing the SQL statement on the false simulated data. In 515, the method may simultaneously include a step of generating a first query result for the false simulated data by executing the SQL statement on the false simulated data, and a step of generating a second query result for the false simulated data by running a generative ML model on natural language input and the false simulated data. In 516, the first query result may have a first subset of tabular data extracted from the false simulated data, and the second query result may have a second subset of tabular data extracted from the false simulated data, and the method may include a step of verifying the validity of the SQL statement based on a comparison between the first subset of tabular data and the second subset of tabular data.

[0097] The embodiments described above may be implemented in hardware, in computer programs executed by a processor, in firmware, or in a combination thereof. Computer programs may be implemented on computer-readable media such as storage media. For example, computer programs may reside in random access memory ("RAM"), flash memory, read-only memory ("ROM"), erasable programmable read-only memory ("EPROM"), electrically erasable programmable read-only memory ("EEPROM"), registers, hard disks, removable disks, compact disk read-only memory ("CD-ROM"), or any other form of storage media known in the art.

[0098] An exemplary storage medium may be coupled to a processor so that the processor can read information from and write information to the storage medium. In an alternative configuration, the storage medium may be integrated with the processor. The processor and storage medium may reside within an application-specific integrated circuit (ASIC). In an alternative configuration, the processor and storage medium may exist as discrete components.

Claims

1. memory; and At least one processor communicatively coupled to the memory The processor comprises: Based on natural language input, A generative machine learning (ML) model is executed on the aforementioned natural language input to generate structured query language (SQL) statements. The SQL statement is executed against the false simulated data to generate a first query result for the false simulated data. The generating ML model is executed on the natural language input and the fake simulated data to generate a second query result for the fake simulated data, and Based on a comparison of the first query result and the second query result, it is determined whether the SQL statement is valid. A device configured in such a way.

2. The apparatus according to claim 1, wherein the at least one processor is configured to determine that the SQL statement is valid if the content included in the first query result includes all the content included in the second query result.

3. The apparatus according to claim 1, wherein the at least one processor is configured to determine that the SQL statement is invalid if the content included in the first query result does not include all the content included in the second query result.

4. The apparatus according to claim 1, wherein the at least one processor is configured to, in response to a determination that the SQL statement is valid, execute the SQL statement on valid data stored in the database to generate a valid query result, and output the valid query result to a software application.

5. The apparatus according to claim 1, wherein the at least one processor is configured to generate the false simulated data by running the generating ML model on the database schema and the data types associated with the SQL statement before executing the SQL statement on the false simulated data.

6. The apparatus according to claim 1, wherein the at least one processor is configured to simultaneously execute the SQL statement on the false simulated data to generate a first query result for the false simulated data, and execute the generating ML model on the natural language input and the false simulated data to generate a second query result for the false simulated data.

7. The apparatus according to any one of claims 1 to 6, wherein the first query result has a first subset of tabular data extracted from the false simulated data, and the second query result has a second subset of tabular data extracted from the false simulated data, wherein at least one processor is configured to verify the validity of the SQL statement based on a comparison between the first subset of tabular data and the second subset of tabular data.

8. The stage following natural language input; The step of running a generative machine learning (ML) model on the aforementioned natural language input to generate a structured query language (SQL) statement; A step of executing the SQL statement on the false simulated data to generate a first query result for the false simulated data; A step of running the generating ML model on the natural language input and the false simulated data to generate a second query result for the false simulated data; and A step in which it is determined whether the SQL statement is valid based on a comparison of the first query result and the second query result. A method for providing this.

9. The method according to claim 8, wherein the determination step includes determining that the SQL statement is valid if the content included in the first query result includes all the content included in the second query result.

10. The method according to claim 8, wherein the determination step includes determining that the SQL statement is invalid if the content included in the first query result does not include all the content included in the second query result.

11. The method according to claim 8, further comprising the steps of: executing the SQL statement on valid data stored in the database to generate valid query results in response to a determination that the SQL statement is valid; and outputting the valid query results to a software application.

12. The method according to claim 8, further comprising the step of generating the false simulated data by running the generated ML model on the database schema and the data types associated with the SQL statement before executing the SQL statement on the false simulated data.

13. The method according to claim 8, wherein the step of executing the SQL statement simultaneously includes the steps of executing the SQL statement on the false simulated data to generate a first query result for the false simulated data, and executing the generated ML model on the natural language input and the false simulated data to generate a second query result for the false simulated data.

14. The method according to any one of claims 8 to 13, wherein the first query result has a first subset of tabular data extracted from the false simulated data, the second query result has a second subset of tabular data extracted from the false simulated data, and the determination step has a step of confirming the validity of the SQL statement based on a comparison of the first subset of tabular data and the second subset of tabular data.

15. A computer program comprising instructions, wherein when the instructions are executed by a processor, the processor: Procedure for receiving natural language input; A procedure for generating structured query language (SQL) statements by running a generative machine learning (ML) model on the aforementioned natural language input; A procedure for executing the SQL statement against false simulated data to generate a first query result for the false simulated data; A procedure for running the generating ML model on the natural language input and the false simulated data to generate a second query result for the false simulated data; and A procedure for determining whether the SQL statement is valid based on a comparison of the first query result and the second query result. A computer program that executes something.

16. The computer program according to claim 15, wherein the determination procedure includes determining that the SQL statement is valid if the content included in the first query result includes all the content included in the second query result.

17. The computer program according to claim 15, wherein the determination procedure includes determining that the SQL statement is invalid if the content included in the first query result does not include all the content included in the second query result.

18. The computer program according to claim 15, wherein the processor is configured to perform the steps of executing the SQL statement on valid data stored in a database to generate valid query results, and outputting the valid query results to a software application, in response to a determination that the SQL statement is valid.

19. The computer program according to claim 15, wherein the processor is configured to perform a procedure to generate the false simulated data by running the generating ML model on the database schema and the data types associated with the SQL statement before executing the SQL statement on the false simulated data.

20. The computer program according to any one of claims 15 to 19, wherein the procedure for executing the SQL statement simultaneously includes the procedure for executing the SQL statement on the false simulated data to generate a first query result for the false simulated data, and the procedure for executing the generated ML model on the natural language input and the false simulated data to generate a second query result for the false simulated data.

Images