Synthetic data generation for query generation

Abstract

The systems, methods, apparatus, and computer-readable storage media described herein provide techniques for generating synthetic data for use in query generation. In one aspect, pairs including natural language (NL) queries and query language (QL) queries, along with predictive catalog information, are used to prompt a large language model (LLM) to generate augmented pairs as variants of those pairs. Synthetic data including the augmented pairs is generated. In another aspect, an instruction is received as feedback on a QL query generated by the LLM, and a correction pair is generated based on that instruction and the corresponding NL query. The correction pair includes a corrected QL query and an NL query. The corrected QL query is a syntactically valid transformation of the NL query. The correction pair is determined to meet data storage criteria and is stored as synthetic data in the data storage.

Description

Background Technology

[0001] Queries written in a query language can be used to perform database operations, such as retrieving and / or transforming records in a database. Query language queries rely on two sources of knowledge: language knowledge and database knowledge. Systems used to generate query language queries may possess parameterized knowledge of that language. For example, a system may utilize a generative artificial intelligence (AI) model trained on a large information corpus to generate query language queries. This large information corpus may or may not specifically contain database knowledge.

[0002] Generative AI models can exhibit "illusions," meaning they generate incorrect or misleading results. Some implementations of query language generation employ preprocessing and post-processing techniques to validate and / or correct queries generated by the generative AI model. Summary of the Invention

[0003] This summary is provided to introduce selected concepts in a simplified form, which will be further described in the detailed embodiments described below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0004] This document describes embodiments for generating synthetic data used in query language (QL) query generation. For example, in one aspect of this disclosure, a dataset pair comprising a natural language (NL) query and a query language query is obtained. A cue word is generated using this dataset pair and predicted directory information to enable a generative artificial intelligence (AI) model (such as a large language model (LLM)) to generate a variant of the dataset pair. In response to providing the cue word to the LLM, an enhancement pair is received. This enhancement pair includes an enhanced NL query and an enhanced QL query. The enhanced NL query is a variant of the NL query, and the enhanced QL query is a variant of the QL query. In this aspect, synthetic data comprising the enhancement pair is generated.

[0005] In another embodiment of this first aspect, directory information is predicted based on the similarity between dataset pairs (or embeddings of dataset pairs) and portions of the database (or embeddings of these portions).

[0006] In another embodiment of this first aspect, the generation of enhancement pairs is performed iteratively to generate multiple enhancement pairs.

[0007] In another aspect of this disclosure, an instruction is received regarding negative feedback to a QL query generated by a generative AI model based on an NL query. A correction pair is generated based on the instruction and the NL query. The correction pair includes the NL query and a corrected QL query. The corrected QL query is a syntactically valid transformation of the NL query. It is determined whether the correction pair meets the criteria for synthetic data storage. If the correction pair meets the criteria, it is stored as synthetic data in the synthetic data storage.

[0008] In another embodiment of this second aspect, a correction pair is generated based on session telemetry.

[0009] In another embodiment of this second aspect, a generative AI model is used to generate the correction pair.

[0010] In another embodiment of this second aspect, a calibration pair is generated by iteratively utilizing a generative AI model.

[0011] In another embodiment of any aspect, the syntax of the enhanced or corrected pair, its similarity to existing pairs, its coverage of the database, and / or the consistency of the transformation are evaluated.

[0012] In another embodiment of any aspect, the language conversion engine generates cue words based on natural language input and enhancement pairs and / or correction pairs. These cue words are then provided to a generative AI model to generate a QL query. Attached Figure Description

[0013] The accompanying drawings, which are incorporated herein and form part of this specification, illustrate embodiments and, together with this description, further serve to explain the principles of these embodiments and to enable those skilled in the art to implement and use these embodiments.

[0014] Figure 1 A block diagram of a system for query generation according to an example embodiment is shown.

[0015] Figure 2 A block diagram of a system for generating synthetic data according to an example embodiment is shown.

[0016] Figure 3 A flowchart of a process for generating synthetic data according to an example embodiment is shown.

[0017] Figure 4 A block diagram of a system for generating synthetic data according to another example embodiment is shown.

[0018] Figure 5 A flowchart of a process for receiving an embedding is shown according to an example embodiment.

[0019] Figure 6A flowchart of a process for predicting directory information according to an example embodiment is shown.

[0020] Figure 7 A block diagram of a system for generating synthetic data according to another example embodiment is shown.

[0021] Figure 8 A flowchart of a process for generating synthetic data according to another example embodiment is shown.

[0022] Figure 9 A flowchart is shown for a process of filtering enhancement pairs from a set of enhancement pairs according to an example embodiment.

[0023] Figure 10 A flowchart of a process for filtering enhancement pairs from a set of enhancement pairs is shown according to another example embodiment.

[0024] Figure 11 A flowchart of a process for filtering enhancement pairs from a set of enhancement pairs is shown according to another example embodiment.

[0025] Figure 12 A flowchart of a process for iteratively generating synthetic data, according to an example embodiment, is shown.

[0026] Figure 13 A block diagram of a system for iteratively generating synthetic data according to another example embodiment is shown.

[0027] Figure 14 A flowchart of a process for iteratively generating synthetic data according to another example embodiment is shown.

[0028] Figure 15 An example tree 1500 (“tree 1500” herein) of an enhancement pair according to an embodiment is shown.

[0029] Figure 16 A block diagram of a system for generating synthetic data according to an example embodiment is shown.

[0030] Figure 17 A flowchart of a process for generating synthetic data according to an example embodiment is shown.

[0031] Figure 18 A block diagram of a system for generating synthetic data according to another example embodiment is shown.

[0032] Figure 19 A flowchart of a process for generating a correction pair according to an example embodiment is shown.

[0033] Figure 20 A flowchart of a process for generating a correction pair according to another example embodiment is shown.

[0034] Figure 21 A flowchart of a process for iteratively generating correction pairs according to an example embodiment is shown.

[0035] Figure 22 A flowchart of a process for determining the eligibility of a query, according to another example embodiment, is shown.

[0036] Figure 23 A block diagram of a system for evaluating the consistency of calibration pairs according to an example embodiment is shown.

[0037] Figure 24 A flowchart of a process for evaluating the consistency of a calibration pair, according to an example embodiment, is shown.

[0038] Figure 25 A block diagram of a system for generating queries according to an example embodiment is shown.

[0039] Figure 26 A flowchart of a process for generating a query, according to an example embodiment, is shown.

[0040] Figure 27 A flowchart illustrating a process for preprocessing prompt words for a generative AI model, according to an example embodiment, is shown.

[0041] Figure 28 A block diagram of an example computing environment in which embodiments can be implemented is shown.

[0042] The subject matter of this application will now be described with reference to the accompanying drawings. In the drawings, the same reference numerals denote the same or similarly functional elements. Furthermore, the leftmost numeral(s) of the reference numeral(s) identifies the drawing in which that reference numeral(s) first appears. Detailed Implementation

[0043] I. Introduction The following detailed description discloses numerous exemplary embodiments. The scope of this patent application is not limited to the disclosed embodiments, but also covers combinations of the disclosed embodiments and modifications thereof. Note that any section / subsection headings provided herein are not intended to be limiting. Embodiments are described throughout this document, and any type of embodiment may be included under any section / subsection. Furthermore, embodiments disclosed in any section / section may be combined in any way with any other embodiments described in the same section / section and / or different sections / sections.

[0044] II. Examples Used for Query Generation Embodiments of this disclosure relate to query generation, such as query language queries (e.g., Kusto Query Language (KQL) queries, Structured Query Language (SQL) queries, etc.). Query language queries (also referred to herein as “QL queries”) are used to perform database operations, such as, but not limited to, retrieving and / or transforming records in a database. For example, an application (or a user utilizing an application or computing device) can provide a QL query to be performed against a database to retrieve and manipulate data in the database. According to embodiments, QL queries rely on knowledge of the query language and knowledge of the database being queried. In some implementations of query generation, a natural language to query language engine (also referred to herein as a “language conversion engine”) is used to facilitate the generation of QL queries for database execution. For example, a user or application provides a query in natural language (i.e., a language of common speaking and / or writing) to the language conversion engine. The language conversion engine converts the provided query (also referred to herein as a “natural language query” or “NL query”) into a QL query suitable for execution against the database. In this way, the language conversion engine simplifies the interaction between the user or application wishing to access or manipulate data in the database and the database.

[0045] In some implementations of QL query generation, generative artificial intelligence (AI) models are utilized to generate QL queries. Generative AI models are models that generate complex, coherent, and / or original content. For example, generative AI models can create complex sentences, lists, ranges, data tables, images, papers, etc. An example of a generative AI model is a language model. For example, some embodiments described herein utilize Large Language Models (LLMs). An LLM is a language model with a large number of model parameters (e.g., weights and biases learned by the model during training). LLMs are (pre)trained using self-supervised learning and / or semi-supervised learning. Some implementations of LLMs are transformer-based LLMs (e.g., the Generative Pre-trained Transformer (GPT) model family). A transformer is a neural network architecture that relies on a self-attention mechanism to transform an input embedding sequence into an output embedding sequence (e.g., without relying on convolutional or recurrent neural networks). Additional details on transformer-based LLMs (and generative AI models in general) can be found in [reference needed]. Figure 28 And described elsewhere in this article.

[0046] Techniques utilizing generative AI models can experience a “hallucination,” where the model generates incorrect or misleading results. Furthermore, generative AI models can generate QL queries that are grammatically valid but fail to meet the expectations of the user or the calling application(s). In particular, some implementations of generative AI models may struggle to generate QL queries if labeled data is scarce (e.g., when processing new data sources or customer queries, also known as a “cold start”). Some implementations of language translation engines that utilize generative AI models to generate QL queries leverage synthetic data (also known as “few-shot examples”) to enhance the prompting process. However, existing synthetic data may be inconsistent with the database constrained by the user query. Even if the synthetic data is indeed aligned with the database, the amount of synthetic data aligned with the user query may be limited.

[0047] In one aspect of this disclosure, the methods, systems, and computer-readable storage media described herein provide techniques for generating synthetic data that is efficiently aligned with user queries. For example, in one embodiment, a dataset pair comprising an NL query and a QL query is obtained. The dataset pair and prediction catalog information are used to generate prompt words to enable a generative AI model (such as an LLM) to generate variants of the dataset pair. The prediction catalog information includes a description of the database, a description of the data stored therein, and / or a description of the structure and / or grouping of stored data similar to (e.g., semantically similar) the dataset pair. In response to providing prompt words to the LLM, an enhancement pair is received. The enhancement pair includes an enhanced NL query and an enhanced QL query. The enhanced NL query is a variant of the NL query, and the enhanced QL query is a variant of the QL query. In this respect, synthetic data comprising the enhancement pair is generated.

[0048] In another aspect of this disclosure, the methods, systems, and computer-readable storage media described herein provide techniques for generating synthetic data consistent with user queries based on user feedback. For example, in one embodiment, an instruction for negative feedback on a QL query generated by a generative AI model based on an NL query is received. A correction pair is generated based on the instruction and the NL query. The correction pair includes the NL query and a corrected QL query. The corrected QL query is a syntactically valid transformation of the NL query. Correction pairs can be generated in various ways. For example, in some embodiments, correction pairs are generated based on a QL query performed against a database. In other embodiments, correction pairs are generated by providing cue words to the generative AI model. In either case, it is determined whether the correction pair meets the criteria for synthetic data storage. If the correction pair meets the criteria, the correction pair is stored as synthetic data in the synthetic data storage.

[0049] Systems, devices, and apparatuses can be configured in various ways to generate synthetic data and / or QL queries based on natural language. For example, Figure 1A block diagram of a query generation system 100 according to an example embodiment is shown. System 100 includes a computing device 102, a conversion server 104, an embedded server 106, a synthetic data server 108, a model server 110, an engine server 112, a database 114, and a storage device 116. The computing device 102, conversion server 104, embedded server 106, synthetic data server 108, model server 110, engine server 112, database 114, and storage device 116 are communicatively coupled via a network 118. In the example, network 118 includes one or more networks, such as a local area network (LAN), a wide area network (WAN), an enterprise network, the Internet, etc. In the example, network 118 includes one or more wired and / or wireless components. The features of system 100 are described in detail below.

[0050] Database 114 is configured to store data. Examples of database 114 include, but are not limited to, unstructured databases (e.g., binary large object (blob) storage devices), structured databases (e.g., SQL databases), and semi-structured databases. In implementations, database 114 includes any amount of data organized in various ways. For example, such as Figure 1 As shown, database 114 includes storage for the corresponding datasets 138A-138. n Table 136A-136 n Table 136A-136 n Each table in the table contains data organized as 138A-138. n One or more columns of the corresponding data. According to the embodiment, Tables 136A-136... n The tables in the table are grouped into "clusters" (for brevity, Figure 1 (Not shown in the image). According to an embodiment, database 114 is implemented as cloud-based storage (e.g., cloud-based data lake storage, cloud-based file system, cloud-based database, etc.). In this case, database 114 is stored by one or more servers on a networked server infrastructure (for simplicity, ...). Figure 1 (Not shown in the text)

[0051] Storage device 116 stores data used and / or generated by computing device 102, conversion server 104, embedding server 106, synthetic data server 108, model server 110, engine server 112 and / or components thereof and / or services executed thereon. For example, such as... Figure 1As shown, storage device 116 stores pair data 142 and synthetic data 144. Pair data 142 includes dataset pairs of NL queries and QL queries. For example, the dataset pairs of data 142 represent the transformation from NL queries to QL queries. In the example, dataset pairs are obtained from manually generated pairs, evaluations of executed QL queries, analyst surveys, user feedback, and / or any other suitable source for mapping natural language input to QL queries. Synthetic data 144 represents synthetic data generated by synthetic data server 108 (or a service performed by it), as described elsewhere herein. In the example, synthetic data 144 includes synthetic NL and QL query pairs. In this case, the synthetic pairs are pairs generated by synthetic data server 108 (or a service performed by it), as described elsewhere herein. According to embodiments, all or part of synthetic data 144 is a subset of pair data 142.

[0052] like Figure 1 As shown, storage device 116 is external to computing device 102, conversion server 104, embedded server 106, synthetic data server 108, model server 110, engine server 112, and database 114. In alternative example embodiments, all or part of storage device 116 is internal to computing device 102, conversion server 104, embedded server 106, synthetic data server 108, model server 110, engine server 112, and / or database 114. According to an embodiment, storage device 116 is a remote storage device (e.g., web storage, blob storage, networked file system, cloud storage, etc.) accessible via network 118.

[0053] In this example, computing device 102 is any type of fixed or mobile processing device, including but not limited to desktop computers, servers, mobile or handheld devices (e.g., tablets, personal digital assistants (PDAs), smartphones, laptops, etc.), Internet of Things (IoT) devices, etc. According to an embodiment, computing device 102 is associated with users (e.g., individual users, user groups, organizations, home users, customer users, employee users, administrator users (e.g., service team users, developer users, management users, etc.)). Computing device 102 is configured to execute application 120. According to an embodiment, application 120 enables users to interface with conversion server 104, embedding server 106, synthetic data server 108, model server 110, engine server 112, database 114, and / or storage device 116.

[0054] The conversion server 104, embedded server 106, synthetic data server 108, model server 110, and engine server 112 are network-accessible servers (or other types of computing devices). According to an embodiment, one or more of the conversion server 104, embedded server 106, synthetic data server 108, model server 110, and engine server 112 are incorporated into a set of network-accessible servers (e.g., a cloud-based environment, an enterprise network server set, etc.). Furthermore, as... Figure 1 As shown, each of the conversion server 104, embedded server 106, synthetic data server 108, model server 110, and engine server 112 is a single server or other computing device. In an alternative example embodiment, any one of the conversion server 104, embedded server 106, synthetic data server 108, model server 110, and engine server 112 is implemented across multiple servers or computing devices (e.g., as a distributed service). Each of the conversion server 104, embedded server 106, synthetic data server 108, model server 110, and engine server 112 is configured to perform services and / or store data. For example, as... Figure 1 As shown, conversion server 104 is configured to execute language conversion engine 146 and embedded model interface 140, embedded server 106 is configured to execute embedded model 128, synthetic data server 108 is configured to execute synthetic data generator 130, model server 110 is configured to execute generative AI model 132, and engine server 112 is configured to execute database engine 134. According to an embodiment, application 120 interfaces with language conversion engine 146, embedded model 128, generative AI model 132, and / or database engine 134 via network 118.

[0055] Application 120 includes an application configured to generate QL queries using language conversion engine 146, generate embeddings using embedding model 128, generate synthetic data using synthetic data generator 130, and / or cause QL queries to be executed against database 114 (e.g., using database engine 134). For example, application 110 according to an embodiment is a developer or management application for generating synthetic data to be used in QL query generation. For example, an example of such an embodiment of application 120 causes synthetic data generator 130 to generate synthetic data 144.

[0056] In other examples, application 120 is an application used to analyze network threats, benchmark data, analyze customer data, and / or any other type of application suitable for performing queries against database 114. In this case, an embodiment of such application 120 sends a request to the language conversion engine 146 to query the database, causing a QL query to be generated. According to an embodiment, the request includes an NL query. In the example, the NL query takes the form of a question, request, or some other form of natural language input that causes the language conversion engine 146 to generate a QL query, as described elsewhere herein. According to an embodiment, application 120 receives the QL queries generated by the language conversion engine 146 and sends them to the database engine 134 for execution. Alternatively, the QL queries generated by the language conversion engine 146 are automatically provided to the database engine 134.

[0057] In other examples (or in the example where application 120 is used to send a request to generate a QL query), application 120 is an application used to generate feedback for the QL query generated by language conversion engine 146. In this case, an embodiment of application 120 provides feedback to synthetic data generator 130 for generating synthetic data based on the feedback, as described elsewhere herein.

[0058] Embedding model 128 is a model configured to generate embeddings for machine learning. The embeddings generated by embedding model 128 are information-dense representations of the semantic meaning of an input (e.g., a piece of text). For example, according to an embodiment, the embeddings are vectors of floating-point numbers such that the distance between two embeddings in the vector space is related to the semantic similarity between two inputs in their original format (e.g., text format). As an example, if two texts are similar, their vector representations should also be similar. In this way, the embeddings generated by embedding model 128 provide a representation of data that can be used by the system described herein to perform various functions associated with the data represented by the embeddings. For example, according to an embodiment, synthetic data generator 130 utilizes embeddings to predict catalog information (e.g., information about...). Figure 4-6 (and as described elsewhere in this article).

[0059] Synthetic data generator 130 is configured to generate synthetic data. According to one embodiment, synthetic data generator 130 generates synthetic data in response to a request for synthetic data from language conversion engine 146. According to another embodiment, synthetic data generator 130 generates synthetic data when invoked by an application developed by the developer of synthetic data generator 130, language conversion engine 146, and / or database 114. According to another embodiment, synthetic data generator 130 generates synthetic data periodically (once a week, once a month, once a quarter, etc.) and / or in other routine ways (e.g., after the database (e.g., database 114) is updated, as part of the maintenance of language conversion engine 146, etc.)...

[0060] The language conversion engine 146 is configured to convert natural language input (e.g., NL queries) into QL queries. Figure 1 As shown, the language conversion engine 146 is a service executed by the conversion server 104. Alternatively, one or more components of the language conversion engine 146 may be provided by application 120 (or another application executed on computing device 102; for simplicity, ...). Figure 1 (Not shown in the image) Implementation. For example... Figure 1 As shown, the language conversion engine 146 includes a preprocessor 122, a prompter 124, and a postprocessor 126. The preprocessor 122 includes logic for: receiving a request to generate a QL query, refining a mode, selecting synthetic data to include in the prompt, determining additional context to include in the prompt for the generative AI model 132, and / or performing any other operations on preprocessed information to generate the prompt for the generative AI model 132 so that the generative AI model 132 generates a QL query. According to an embodiment, the preprocessor 122 includes an interface for communicating with the embedded model 128 via a network 118. Additional details regarding the preprocessor 122 are as follows: Figure 25-27 And described elsewhere in this article.

[0061] The prompter 124 includes logic for providing prompt words to the generative AI model 132 to enable the generative AI model 132 to generate QL queries. According to one embodiment, the prompter 124 provides the prompt words to the generative AI model 132 as an application programming interface (API) call to the generative AI model 132. According to another embodiment, the prompter 124 includes an interface for communicating with the generative AI model 132 via a network 118. Additional details regarding the prompter 124 can be found in [reference needed]. Figure 25 and Figure 26 And described in other parts of this article.

[0062] Postprocessor 126 includes logic for: parsing QL queries, repairing QL queries, providing responses on behalf of the generative AI model, causing QL queries to be executed (e.g., by providing QL queries to database engine 134), and / or performing any other operations regarding the post-processed QL queries generated by generative AI model 132. According to embodiments, postprocessor 126 includes corresponding interfaces for communicating with embedded model 128, generative AI model 132, and / or database engine 134 via network 118. Additional details regarding postprocessor 126 are available for reference. Figure 25 And described elsewhere in this article.

[0063] Generative AI model 132 is configured to generate QL queries based on received prompts. In this example, generative AI model 132 is any type of generative AI model capable of generating QL queries, generating NL and QL query pairs, and / or generating corrected queries based on prompts received from prompter 124. According to an embodiment, generative AI model 132 is an LLM. In this example, generative AI model 132 is trained using public information (e.g., information collected and / or cleaned from the internet) and / or data stored by the administrator of model server 110 (e.g., stored in the memory of model server 110 and / or in memory accessible to model server 110). According to an embodiment, generative AI model 132 is a “ready-made” model trained to generate complex, coherent, and / or original content based on (e.g., any) prompts. In an alternative example embodiment, generative AI model 132 is a specialized model trained to generate QL queries, natural language and QL query pairs, and / or corrected queries. According to the embodiment, generative AI model 132 and embedded model 128 are the same model. Additional details regarding the operation and training of generative AI models (such as generative AI model 132) are described in Section VI of this disclosure and elsewhere herein.

[0064] Database engine 134 is configured to execute queries against a database (e.g., database 114) to generate query results. In some embodiments, database engine 134 implements query optimization techniques. Figure 1 As shown, database engine 134 is executed by engine server 112. Alternatively, database engine 134 is implemented by an application (e.g., application 120) executed by computing device 102. In another alternative embodiment, database engine 134 is implemented as a component of language conversion engine 146 (e.g., as a subcomponent of post-processor 126 or as a separate component of language conversion engine 146).

[0065] Therefore, System 100 has been described regarding the generation of synthetic data for query generation, the generation of QL queries, and the execution of queries against the database. Additional details regarding the generation of synthetic data and the provision of cue words to the generative AI model to enable the generative AI model to generate QL queries are described in the following sections (and elsewhere in this document).

[0066] III. Examples of Synthetic Data Generation Based on Dataset Pairs Embodiments of the synthetic data generator 130 are configured to generate synthetic data. In examples, a language conversion engine (such as language conversion engine 146) selects the synthetic data generated by the synthetic data generator 130 to include in prompts for the generative AI model 132, enabling the generative AI model 132 to generate QL queries. In these examples, the selected synthetic data provides additional context for QL queries generated through example transformations of natural language input to QL queries. In this way, embodiments improve the quality of QL queries generated by the generative AI model 132 and reduce the likelihood of the generative AI model 132 experiencing hallucinations during the query generation process.

[0067] Examples of synthetic data generator 130 are configured in various ways to generate synthetic data. For example, synthetic data generator 130 according to one or more embodiments is configured to generate synthetic data based on dataset pairs of data 142. To better understand such embodiments, this document describes... Figure 2 . Figure 2 A block diagram of a system 200 for generating synthetic data according to an example embodiment is shown. Figure 2 As shown, system 200 includes storage device 116 (storing paired data 142 and synthesized data 144), synthesized data generator 130, and generative AI model 132, as per [reference to...]. Figure 1 As described. Also, for example... Figure 2 As shown, the synthetic data generator 130 includes a prompt word generator 202 and a synthetic data post-processor 204, each of which is implemented as a component and / or sub-service of the synthetic data generator 130. To better understand the operation of system 200, regarding... Figure 3 describe Figure 2 . Figure 3 A flowchart 300 is shown illustrating a process for generating synthetic data according to an example embodiment. According to the embodiment, the synthetic data generator 130 operates according to flowchart 300. Not all steps of flowchart 300 need to be performed in all embodiments. Based on the following... Figure 2 and 3 Other structural and operational embodiments will be apparent to those skilled in the art as described herein.

[0068] Flowchart 300 begins at step 302. In step 302, a dataset pair including a first natural language query and a first query language query is obtained. For example, Figure 2 The prompt word generator 202 obtains (or otherwise receives) dataset pair 206 from data pair 142. Dataset pair 206 includes a first natural language query and a first QL query. In the example, dataset pair 206 is a manually generated dataset pair, a dataset pair obtained from published data, or a dataset pair previously generated by synthetic data generator 130 (or by another synthetic data generator). Non-limiting examples of dataset pair 206 are shown in Table 1 below:

[0069] In step 304, the dataset pairs and the first prediction catalog information are used to generate a first cue word so that the LLM generates a variant of the dataset pairs. For example, Figure 2 The prompt word generator 202 generates prompt word 210 based on dataset pair 206 and prediction catalog information 208, and provides prompt word 210 to generative AI model 132 to cause generative AI model 132 to generate variants of dataset pair 206. According to an embodiment, prompt word 210 includes instructions to include a specific number of variants of dataset pair 206 (e.g., one variant, two variants, dozens of variants, and / or any other number of variants). In the example, the number of variants indicated in prompt word 210 is predetermined based on the configuration of prompt word generator 202, determined based on the number of dataset pairs from which synthetic data generator 130 will generate synthetic data (e.g., if there are more than a threshold number of dataset pairs in the pair queue from which synthetic data is generated, the prompt word generator 202 according to the embodiment reduces the requested number of variants), and based on instructions provided to synthetic data generator 130 to cause synthetic data generator 130 to generate synthetic data. Figure 2 The prompt word generator 202 may determine the number of variants based on the coverage of existing synthetic data (e.g., if the coverage is sparse, the prompt word generator 202 requests additional variants in the example), or based on the available storage space in storage device 116 (e.g., if the size of synthetic data 144 is close to the limit, the prompt word generator 202 reduces the number of requested variants).

[0070] like Figure 2As shown, the prompt word generator 202 receives predicted directory information 208. Predicted directory information 208 includes a description of the database, a description of the data stored therein, and / or a description of the structure and / or grouping of stored data similar to (e.g., semantically similar) the dataset pair 206. According to an embodiment, predicted directory information 208 includes most or all of the directory information of database 114. Alternatively, predicted directory information 208 includes a portion of the directory information of database 114. In an example of this alternative, predicted directory information 208 is included in the pair of data 142 (e.g., mapped to dataset pair 206). In another example of this alternative, predicted directory information 208 is communicatively determined by a subcomponent of synthetic data generator 130 and / or another component of system 200, for brevity... Figure 2 Not shown in the text. For example, regarding... Figure 4 Examples of components of the synthetic data generator 130 configured to predict catalog information 208 are described elsewhere in this document.

[0071] In step 306, in response to providing a first prompt word to the LLM, a first enhancement pair is received, comprising a first enhanced natural language query and a first enhanced query language query, wherein the first enhanced natural language query is a variant of the first natural language query, and the first enhanced query language query is a variant of the first query language query. For example, the synthetic data post-processor 204 receives enhancement pair 212. Enhancement pair 212 comprises an enhanced NL query as a variant of the NL query of dataset pair 206 and an enhanced QL query as a variant of the QL query of dataset pair 206. Figure 2 As shown, the synthetic data post-processor 204 receives a single enhancement pair from the generative AI model 132. Alternatively, the synthetic data post-processor 204 receives multiple enhancement pairs (e.g., a set of enhancement pairs) from the generative AI model 132 based on cue words 210. For example, suppose the dataset pairs shown in Table 1 are provided to the generative AI model 132. As a continuing non-limiting example, according to an embodiment, the synthetic data post-processor 204 receives enhancement pairs as shown in Table 2 below:

[0072] As shown in Table 2, in this unrestricted example, generative AI model 132 generates three augmentation pairs from the dataset pairs in Table 1.

[0073] In step 308, synthetic data including the first enhancement pair is generated. For example, Figure 2 The post-processor 204 generates synthesized data 214. Synthetic data 214 includes the enhancement pairs (or multiple enhancement pairs) received in step 306. According to embodiments, and as per [reference to...], Figure 7-11(And elsewhere in this document) further discusses how the synthetic data post-processor 204 filters one or more augmentation pairs from the augmentation pairs generated by the generative AI model 132 to generate synthetic data 214. As... Figure 2 As shown, the composite data post-processor 204 stores the composite data 214 as composite data 144 in the storage device 116. Alternatively or additionally, the composite data post-processor 204 sends... Figure 1 The language conversion engine 146 provides synthetic data 214 (for example, for generating queries).

[0074] As described herein, in the examples, synthetic data generator 130 is configured in various ways to generate synthetic data (e.g., synthetic data 144). For example, as per [reference to...] Figure 3 The flowchart 300, step 304, is described. Figure 2 The cue word generator 202 generates cue words based on predicted directory information (and dataset pairs) and provides these cue words to the generative AI model 132, enabling the generative AI model 132 to generate (multiple) augmentation pairs for generating synthetic data. In some examples, the synthetic data generator 130 utilizes an embedding model to generate embeddings and predict directory information while generating synthetic data. Examples of the synthetic data generator 130 are configured in various ways to generate embeddings and / or predict directory information. For example, Figure 4 A block diagram of a system 400 for generating synthetic data according to another example embodiment is shown. Figure 4 As shown, system 400 includes storage device 116, embedded model 128, synthetic data generator 130, and generative AI model 132, as per [reference to...]. Figure 1 As described. Also, for example... Figure 4 As shown, in the example, the synthetic data generator 130 includes a cue word generator 202 and a synthetic data post-processor 204, as described above. Figure 2 As described, each of the queue 402, the embedding model interface 404, and the directory predictor 406 is implemented as a component and / or sub-service of the synthetic data generator 130. According to an embodiment, the queue 402 is configured to store dataset pairs awaiting processing by the embedding model interface 404 and / or the prompt word generator 202. In an example, the embedding model interface 404 is configured to generate embeddings using the embedding model 128. In an example, the directory predictor 406 is configured to predict directory information associated with a query.

[0075] For example Figure 4 As shown, storage device 116 stores information such as... Figure 1The description includes data 142 and synthetic data 144, as well as data catalog 418. Data catalog 418 includes a description of database 114, a description of the data stored therein, and / or a description of the structure and / or grouping (e.g., clusters, tables, columns, etc.) of the stored data, also referred to herein as "catalog information." In this example, data catalog 418 is a "source" of catalog information for database 114. Examples of data catalog 418 include, but are not limited to, product information describing database 114 and / or the data stored therein, indexes of database 114, descriptions of code associated with database 114, and / or any other type of description suitable for determining the embedding of database 114, as described elsewhere herein and / or as will be understood by one of ordinary skill in the art benefiting from this disclosure. According to one embodiment, data catalog 418 includes a single source of catalog information for database 114. According to another embodiment, data catalog 418 includes multiple sources of catalog information for database 114. For example, in a non-limiting example, data catalog 418 includes separate sources for different parts of database 114 (e.g., different clusters of data catalog 418, different subgroups of data catalog 418, etc.).

[0076] Figure 4 Embodiments of the synthetic data generator 130 operate in various ways to obtain or otherwise receive embeddings and / or database embeddings. To better understand the operation of the system 400 that receives embeddings from the embedding model, regarding... Figure 5 describe Figure 4 . Figure 5 A flowchart 500 for receiving an embedding process according to an example embodiment is shown. According to the embodiment, Figure 4 The synthetic data generator 130 operates according to flowchart 500. Not all steps of flowchart 500 need to be performed in all embodiments. Based on the following... Figure 4 and 5 Other structural and operational embodiments will be apparent to those skilled in the art as described herein.

[0077] Flowchart 500 begins at step 502. In step 502, dataset pairs and / or catalog information are provided to an embedding model, which is configured to generate embeddings based on the input data. For example, Figure 4 The embedding model interface 404 provides an embedding request 414 to the embedding model 128, which is configured to generate embeddings based on input data. In an implementation, the embedding request 414 includes one or more dataset pairs and / or directory information. In an embodiment, the embedding model interface 404 receives dataset pairs from queue 402, data 142, and / or the synthetic data post-processor 204 to include in the embedding request 414. For example, as... Figure 4As shown, queue 402 receives one or more dataset pairs 408 (“dataset pairs 408”) from pair data 142. In this context, queue 402 queues dataset pairs 408 for further processing by embedding model interface 404 and / or cue word generator 202. In the example, dataset pairs 408 are queued in a randomized or deterministic order (e.g., based on their order of storage in pair data 142, according to the LIFO order of pairs received by queue 402, the FIFO order of pairs received by queue 402, and / or any other type of deterministic or predetermined order in which pairs are queued in queue 402, as described elsewhere herein and / or as will be understood by those skilled in the art who benefit from this disclosure). In the example, queue 402 provides the next queued pair 412 (“dataset pair 412”) to embedding model interface 404. Dataset pair 412 is as described in the example. Figure 2 Examples of dataset pair 206 are described. In one example, queue 402 automatically provides dataset pair 412 to the embedded model interface 404. In another example, queue 402 provides dataset pair 412 in response to requests and / or instructions generated by the embedded model interface 404. In yet another instance, queue 402 provides dataset pair 412 on a periodic / routine basis.

[0078] In an embodiment, the embedding model interface 404 receives directory information to be included in the embedding request 414 from the data directory 418. For example, such as... Figure 4 As shown, the embedding model interface 404 receives catalog information 410 from the data catalog 418. Depending on the implementation, the catalog information 410 may represent the entire data catalog 418 or a portion thereof. In an example embodiment, the embedding model interface 404 provides the catalog information 410 along with the dataset pair 412 in the embedding request 414. In an alternative embodiment, the embedding model interface 404 provides the dataset pair 412 and the catalog information 410 to the embedding model 128 in a separate embedding request. For example, in a non-limiting example, the embedding model interface 404 provides the catalog information 410 to the embedding model 128 before receiving the dataset pair 412. In this case, embeddings for the catalog information 410 are generated "offline" relative to the embedding generation of the dataset pair. According to an embodiment, the catalog information 410 represents an updated portion of the data catalog 418 (e.g., a portion of the data catalog 418 that has been added to, modified, deleted from, and / or otherwise modified since the last time an embedding was generated for that portion of the data catalog 418).

[0079] In step 504, embeddings and / or database embeddings are received from the embedding model. For example, Figure 4The directory predictor 406 receives one or more embeddings 416 (“embedded 416” herein) from the embedding model 128. Embedded 416 includes pair embeddings generated by the embedding model 128 based on one or more dataset pairs provided to it (e.g., dataset pair 412 included in embedding request 414, as described with respect to step 502) and / or database embeddings generated by the embedding model 128 based on directory information provided to it (e.g., directory information 410 included in embedding request 414, as described with respect to step 502). Pair embeddings describe the context of the corresponding dataset pair, and database embeddings describe the context of the corresponding portion of database 114. Examples of database embeddings include, but are not limited to, database embeddings describing the context of a database, cluster embeddings describing the context of a corresponding cluster in a database, table embeddings describing the context of a corresponding table in a database, column embeddings describing the context of a corresponding column in a table in a database, and value embeddings describing the context of a corresponding value in a column of a table in a database. As described elsewhere herein, embedded 416 is a vector of floating-point numbers such that the distance between two embeddings in the vector space is correlated with the semantic similarity between two inputs in the original format.

[0080] like Figure 4 As shown, the directory predictor 406 receives embeddings 416 from the embedding model 128. According to an alternative embodiment, the directory predictor 406 receives one or more embeddings from the data store (e.g., storage device 116). For example, assuming, in a non-limiting example, the embedding model 128 generates pair embeddings when dataset pairs are queued in pair queue 402, and generates database embeddings when directory information of the data directory 418 is updated and / or otherwise provided to the embedding model 128 to generate database embeddings. In this case, the embedding model 128 stores the database embeddings in the data store for access by the directory predictor 406 (e.g., later). In another example, the embedding model 128 provides pair embeddings to the directory predictor 406, and (e.g., in response to, simultaneously with, or after providing pair embeddings) the directory predictor 406 receives the stored database embeddings from the data store. In some examples, the embedding model 128 stores pair embeddings in the data store. In this case, the directory predictor 406 is able to receive both pair embeddings and database embeddings at a later time than when the embedding model 128 generates the embeddings.

[0081] Embodiments of the synthetic data generator 130 operate in various ways to predict catalog information. For example, according to an embodiment, Figure 4 The directory predictor 406 predicts directory information (e.g., regarding embeddings received from the embedding model) based on the embeddings received from the embedding model. Figure 5 (as described in flowchart 500). To better understand the operation of the synthetic data generator 130 in predicting directory information based on embeddings, regarding... Figure 6 Further description Figure 4 . Figure 6 A flowchart 600 of a process for predicting directory information according to an example embodiment is shown. According to the embodiment, a directory predictor 406 operates according to flowchart 600. Not all steps of flowchart 600 need to be performed in all embodiments. Based on the following... Figure 4 and 6 Other structural and operational embodiments will be apparent to those skilled in the art as described herein.

[0082] Flowchart 600 begins at step 602. In step 602, a database embedding and a pair of embeddings are received; the database embedding describes the context of a portion of the database, and the pair of embeddings describes the context of a dataset pair. For example, Figure 4 The catalog predictor 406 is about Figure 5 The flowchart 500, step 504, describes any method of receiving the database embedding and / or the embedding.

[0083] In step 604, first predicted directory information is determined based on the similarity between the embedding and a subset of the database embedding. For example, Figure 4 The directory predictor 406 determines predicted directory information 208 based on the similarity between the pair embeddings and the database embeddings. For example, suppose the directory predictor 406 receives pair embeddings corresponding to dataset pair 412. In this example, the directory predictor 406 measures the similarity between the pair embeddings and one or more database embeddings to determine which database embeddings have a similarity to the pair embeddings that satisfies a directory similarity criterion. In an embodiment, the directory similarity criterion specifies a threshold to be satisfied by the similarity measure between the pair embeddings and the database embeddings. According to an embodiment, the directory predictor 406 selects the database embedding most similar to the pair embedding used to determine the predicted directory information. According to another embodiment, the directory predictor 406 selects multiple database embeddings similar to the pair embeddings used to determine the predicted directory information (e.g., all database embeddings that satisfy the directory similarity criterion, the top...). n One (predetermined number) of database embeddings, front n Similar database embeddings (e.g., not necessarily meeting directory similarity criteria, etc.). Example measures of similarity include, but are not limited to, Euclidean distance similarity, cosine similarity, dot product similarity, Jaccard similarity, Levenshtein similarity, and / or any other technique suitable for measuring the similarity between embeddings.

[0084] Therefore, regarding Figure 4 System 400 describes Figure 6 Flowchart 600. (For example...) Figure 4As shown, the directory predictor 406 receives embeddings from the embedding model 128 as a result of the embedding model interface 404 providing dataset pairs and / or directory information to the embedding model 128. In an alternative embodiment, the directory predictor 406 receives embeddings 414 from the embedding model interface 404 as a result of a component other than a subcomponent of the synthetic data generator 130 providing dataset pairs and / or directory information to the embedding model 128. For example, in an example where the dataset pair includes a previously executed QL query from natural language conversion provided to the language conversion engine 146, a component of the language conversion engine 146 provides query information to the embedding model 128 to generate embeddings. In this alternative, the generated embeddings are included in the data pair 142 or otherwise mapped to the dataset pair. In another alternative, embeddings are generated as a result of input provided to the embedding model 128 by a developer. In another example, embeddings of portions of the data directory 418 are included in the storage device 116 and mapped to the corresponding portions of the directory.

[0085] In addition, there has been information regarding Figure 4 System 400 and Figure 6 Flowchart 600 describes an embodiment of a directory predictor as predicting query-related directory information based on the embedding of the query and the embedding of directory information. However, the embodiments described herein are not limited to this. For example, according to an embodiment, the directory predictor of the synthetic data generator 130 evaluates the text of a dataset pair and the text of a portion of the directory information to determine a score (also called a “similarity score”) representing the similarity between the text. Depending on the implementation, such a directory predictor matches characters, words, and / or phrases between the dataset pair and the portion of the directory information. In this example, a higher score indicates a higher match between the text of the dataset pair and the text of that portion of the directory information. In this way, the directory predictor determines the predicted directory information without using the embedding model 128.

[0086] In relation to Figure 4 In the implementation of the described synthetic data generation, after generating the prediction catalog information 208, the process proceeds with... Figure 3 The flowchart 300 continues in the manner described. For example, as... Figure 4As shown, the cue word generator 202 receives a dataset pair 412 from the queue 402 and predicted directory information 208 from the directory predictor 406. The cue word generator 202 generates a cue word 210 from the dataset pair 412 and the predicted directory information 208 (e.g., in the manner described with respect to step 304 of flowchart 300) and provides the cue word 210 to the generative AI model 132 to generate an augmentation pair 212. The synthetic data post-processor 204 receives the augmentation pair 212 (e.g., in the manner described with respect to step 306 of flowchart 300) and generates synthetic data 214 (e.g., in the manner described with respect to step 308 of flowchart 300).

[0087] In some examples, the synthetic data generator 130 generates synthetic data from multiple enhancement pairs. In embodiments, such examples of the synthetic data generator 130 are configured in various ways to generate synthetic data from multiple enhancement pairs. For example, Figure 7 A block diagram of a system 700 for generating synthetic data according to another example embodiment is shown. Figure 7 As shown, system 700 includes a storage device 116 (storing synthetic data 144 and a data directory 418), a synthetic data generator 130 (including a prompt word generator 202, a synthetic data post-processor 204, and a queue 402), and a generative AI model 132, as described above. Figure 1 , 2 4 and other places mentioned herein. Also, Figure 7 As shown, the synthetic data post-processor 204 includes a query parser 702, a similarity estimator 704, a coverage estimator 706, and a filtered pair 708, each of which is implemented as a sub-service / component of the synthetic data post-processor 204. The query parser 702 is configured to parse the query generated by the generative AI model 132 to determine whether the query is valid. Figure 7 As shown, query parser 702 is a sub-service / component of synthetic data post-processor 204. In alternative embodiments, query parser 702 is a separate component of system 700 (e.g., a separate component of synthetic data generator 130 or a component external to synthetic data generator 130). For example, according to an embodiment, query parser 702 is configured to parse augmented queries generated by generative AI model 132 as a result of the operation of synthetic data generator 130, and to parse queries generated by generative AI model 132 as a result of the operation of synthetic data generator 130. Figure 1The query is generated as a result of the operation of the language conversion engine 146. In this alternative embodiment example, the query parser 702 is a subcomponent of the post-processor 126 and / or the post-processor 126 may be accessed in other ways. The similarity evaluator 704 is configured to evaluate the similarity between the augmented pairs and existing dataset pairs (e.g., dataset pairs of data 142 and / or synthetic data 144). The coverage evaluator 706 is configured to evaluate the percentage or portion of the data catalog 418 covered by the augmented pairs and existing dataset pairs. The filtered pairs 708 are augmented pairs filtered by the query parser 702, the similarity evaluator 704, and / or the coverage evaluator 706. In this embodiment, the filtered pairs 708 are generated as synthetic data 214 by the synthetic data generator 130.

[0088] In one embodiment, system 700 generates synthesized data from multiple enhancement pairs. In some embodiments, system 700 filters enhancement pairs from multiple enhancement pairs to generate synthesized data. To better understand this operation of system 700, regarding... Figure 8 Described Figure 7 . Figure 8 A flowchart 800 of a process for generating synthetic data according to another example embodiment is shown. According to the embodiment, system 700 operates according to flowchart 800. Not all steps of flowchart 800 need to be performed in all embodiments. Based on the following... Figure 7 and 8 Other structural and operational embodiments will be apparent to those skilled in the art as described herein.

[0089] Flowchart 800 begins at step 802. In step 802, a first prompt word is provided to the LLM to generate a set of enhancement pairs including a first enhancement pair and a second enhancement pair. The second enhancement pair includes a second enhanced natural language query and a second enhanced query language query, where the second enhanced natural language query is another variant of the first natural language query, and the second enhanced query language query is another variant of the first query language query. For example, Figure 7 The cue word generator 202 provides cue words 210 to the generative AI model 132, enabling the generative AI model 132 to generate multiple enhancement pairs 710 (“enhancement pairs 710”). The cue word generator 202 generates the cue words 210 in any manner described elsewhere herein (e.g., as per [reference to other documents]). Figure 3(As described in step 304 of flowchart 300). Enhancement pairs 710 include any number of enhancement pairs generated by the generative AI model 132 from the prompt 210. In the example, this number is specified either in the prompt 210 or based on the configuration of the generative AI model 132. Each enhancement pair 710 includes an enhanced NL query and an enhanced QL query. The enhanced NL query is a variant of the NL query included in the prompt 210, and the QL query is a variant of the QL query included in the prompt 210. (As...) Figure 7 As shown, the synthesized data post-processor 204 (or a component thereof) receives the enhancement pair 710, and the process continues to step 804.

[0090] In step 804, a second enhancement pair is filtered from the set of enhancement pairs to generate a set of filtered pairs. For example, Figure 7 The synthetic data post-processor 204 (or a component thereof) filters enhancement pairs from enhancement pairs 710 to generate filtered pairs 708. In the example, Figure 7 The synthesized data post-processor 204 operates in various ways to filter one or more enhancement pairs from enhancement pairs 710. In some embodiments, the synthesized data post-processor 204 filters enhancement pairs 710 simultaneously, in batches, or one at a time sequentially. Regarding Figure 9-11 Other examples of filtering enhancement pairs from enhancement pairs 710 are described elsewhere in this article.

[0091] In step 806, synthesized data comprising the filtered set of pairs is generated. For example, synthesized data post-processor 204 generates synthesized data 214 comprising the filtered pair 708. According to an embodiment, synthesized data post-processor 204 stores synthesized data 214 in storage device 116 (e.g., by adding synthesized data 214 to synthesized data 144). According to another embodiment, synthesized data post-processor 204 adds synthesized data 214 to pair queue 402. Further details regarding adding synthesized data to the pair queue can be found in [reference needed]. Figure 12 And described elsewhere in this article.

[0092] As described with respect to step 804, the synthesized data post-processor 204 filters enhancement pairs from enhancement pairs 710 in various ways. For example, according to an embodiment, the synthesized data post-processor 204 filters enhancement pairs from enhancement pairs 710 based on the invalidity of the enhancement pairs. Figure 7 As shown, the synthetic data post-processor 204 includes a query parser 702. The query parser 702 operates in various ways to parse queries. To better understand the synthetic data post-processor 204's use of the query parser 702 to filter augmented pairs, regarding... Figure 9 Further description Figure 7 . Figure 9A flowchart 900 is shown for a process of filtering enhancement pairs from a set of enhancement pairs according to an example embodiment. According to the embodiment, a synthetic data post-processor 204 operates according to flowchart 900. Flowchart 900 is another embodiment of step 804 of flowchart 800. Not all steps of flowchart 900 need to be performed in all embodiments. Based on... Figure 9 Compared to Figure 7 The following description, other structural and operational embodiments will be apparent to those skilled in the art.

[0093] Flowchart 900 begins at step 902. In step 902, a second enhanced QL query is provided to a query parser configured to determine whether the query is valid. For example, Figure 7 The query parser 702 receives the enhancement pair 710 and determines whether the QL query in the enhancement pair 710 is valid. According to an embodiment, the query parser 710 verifies multiple queries at once (e.g., simultaneously or sequentially). In an alternative embodiment, the synthesized data post-processor 204 filters the enhancement pairs 710 sequentially. In this alternative embodiment, the query parser 710 determines whether the enhanced QL query of the enhancement pair is valid. In an embodiment, if the enhanced QL query is invalid, flowchart 900 proceeds to step 904. If the enhanced QL query is valid, such as... Figure 7 As shown, query parser 702 provides validity signal 712 to similarity estimator 704. In an alternative embodiment (e.g., where synthetic data post-processor 204 does not include a similarity estimator), query parser 702 provides validity signal 712 indicating validity (and including enhancement pairs) to coverage estimator 706. In another alternative embodiment (e.g., where synthetic data post-processor 204 does not include a similarity estimator or coverage estimator), query parser 702 includes enhancement pairs in filtered pairs 708, and query parser 702 parses the next enhancement pair.

[0094] In step 904, an indication that the second enhanced QL query is invalid is received. For example, suppose... Figure 7 The query parser 702 determines that the enhanced QL query is invalid. In this case, the query parser 702 filters enhancement pairs from enhancement pairs 710 that include invalid enhanced QL queries (e.g., by discarding the pair) and parses the next enhancement pair of enhancement pairs 710 (e.g., in a manner similar to that described with respect to step 902). By filtering enhancement pairs that include invalid QL queries, the embodiment of the query parser 702 prevents the generation of synthetic data based on invalid QL queries, thereby improving the quality of the synthetic data generated by the synthetic data post-processor 204, and thus improving the quality of the synthetic data generated by the synthetic data post-processor 204. Figure 1 The language conversion engine 146 provides the generative AI model 132 with the quality of prompt words.

[0095] As described in this article, Figure 7 The post-processor 204 operates in various ways to filter enhancement pairs from the enhancement pair set. For example, as... Figure 7 As shown, the synthetic data post-processor 204 includes a similarity estimator 704. The similarity estimator 704 operates in various ways to evaluate the similarity between the augmented pair and existing dataset pairs and / or the synthetic data. To better understand the operation of the similarity estimator 704, see... Figure 10 Further description Figure 7 . Figure 10 A flowchart 1000 is shown for a process of filtering enhancement pairs from a set of enhancement pairs according to another example embodiment. According to the embodiment, a synthetic data post-processor 204 operates according to flowchart 1000. Flowchart 1000 is... Figure 8 Another embodiment of step 804 of flowchart 800. Note that flowchart 1000 does not need to be executed in all embodiments. Based on Figure 10 Compared to Figure 7 The following description, other structural and operational embodiments will be apparent to those skilled in the art.

[0096] Flowchart 1000 includes step 1002. In step 1002, it is determined that the similarity between the second augmentation pair and at least one of the following satisfies a pair similarity criterion: a first dataset pair, a first augmentation pair, or another pre-existing dataset pair. For example, similarity estimator 704 receives a validity signal 712 (including a validated augmentation pair 710) and one or more dataset pairs 714 (“dataset pair 714” herein). Similarity estimator 704 determines whether the similarity between the augmentation pair and any dataset pair 714 satisfies the pair similarity criterion. According to an embodiment, the pair similarity criterion is a threshold that the similarity metric between the augmentation pair and the dataset pair in the dataset pair 714 must satisfy. According to an embodiment, similarity estimator 704 measures the similarity between the text of the augmentation pair and the text of the dataset pair to determine whether the similarity satisfies the pair similarity criterion. In an alternative embodiment, similarity estimator 704 measures the similarity between the pair embeddings of the augmentation pair and the pair embeddings of the dataset pair. In this case, similarity estimator 704 uses the embeddings to determine whether the semantic similarity between the augmentation pair and the dataset pair satisfies the pair similarity criterion. In some embodiments, the pair embeddings of the dataset pairs are stored in storage device 116 (e.g., included in synthetic data 144 or stored separately from synthetic data 144). In other embodiments, similarity estimator 704 provides the dataset pairs to an embedding model (e.g., embedding model 128). Figure 7 (Not shown in the image) to generate pair embeddings for dataset pairs. According to an embodiment, the similarity evaluator 704 provides the augmented pairs to the embedding model to generate pair embeddings for the augmented pairs.

[0097] like Figure 7 As shown, the similarity estimator 704 receives dataset pairs 714 from the synthetic data 144. According to one embodiment, the similarity estimator 704 receives all dataset pairs from the synthetic data 144 as dataset pairs 714. According to another embodiment, the similarity estimator 704 receives one dataset pair 714 at a time, determines whether the similarity between that pair and the augmented pair satisfies a similarity criterion, and if so, filters augmented pairs from the augmented pairs 710; otherwise, it receives the next dataset pair from the dataset pairs 714. In this embodiment, the process continues until augmented pairs are filtered from the augmented pairs 710 or the similarity between all dataset pairs 714 and the augmented pairs has been evaluated. According to another embodiment, the similarity estimator 704 uses the augmented pair (e.g., an embedding describing its context) as an index to search the synthetic data 144 for one or more dataset pairs that have a similarity to the augmented pair that satisfies the similarity criterion.

[0098] In an embodiment, if the similarity evaluator 704 determines that a similarity criterion is met (i.e., the enhanced pair is (e.g., too) similar to existing dataset pairs), the similarity evaluator 704 filters enhanced pairs from enhanced pairs 710 and evaluates the next enhanced pair. For example, as Figure 7 As shown, the similarity evaluator 704 provides the query parser 702 with an indication 724 indicating that the enhancement pairs are filtered (or will be filtered), and the query parser 702 parses the next enhancement query. In an alternative embodiment (e.g., where the synthetic data post-processor 204 does not utilize the query parser), the similarity evaluator 704 evaluates the next enhancement query. By filtering enhancement pairs similar to existing dataset pairs, the synthetic data generator 130, including the similarity evaluator 704, generates different synthetic data for the database. Furthermore, by reducing overlap in dataset pairs, embodiments of the similarity evaluator 704 reduce the storage space consumed by the synthetic data 144.

[0099] In an embodiment, if the similarity evaluator 704 determines that a pair of similarity criteria is not met, the similarity evaluator 704 provides a dissimilarity signal 716 to the coverage evaluator 706. In an example, the dissimilarity signal 716 includes enhanced pairs and / or indicates that the similarity between the enhanced pair and the existing pair does not meet the pair of similarity criteria. In an alternative embodiment (e.g., where the synthetic data post-processor 204 does not include a coverage evaluator), the similarity evaluator 704 includes enhanced pairs in the filtered pairs 708, and the next enhanced pair is processed (e.g., resolved by the query parser 702 and / or has the similarity evaluated by the similarity evaluator 704).

[0100] As described in this article, Figure 7 The post-processor 204 operates in various ways to filter enhancement pairs from the enhancement pair set. For example, as... Figure 7 As shown, the synthetic data post-processor 204 includes a coverage evaluator 706. The similarity evaluator 704 operates in various ways to evaluate the similarity between enhanced pairs and portions of database 114 that have been covered by existing dataset pairs and / or synthetic data, and / or portions of database 114 that have not been covered by existing pairs or synthetic data. To better understand the operation of the coverage evaluator 706, regarding... Figure 11 Further description Figure 7 . Figure 11 A flowchart illustrating a process for filtering enhancement pairs from a set of enhancement pairs according to another example embodiment is shown. According to the embodiment, the synthetic data post-processor 204 operates according to flowchart 1100. Note that flowchart 1100 does not need to be executed in all embodiments. Figure 11 Compared to Figure 7 The following description, other structural and operational embodiments will be apparent to those skilled in the art.

[0101] Flowchart 1100 includes step 1102. In step 1102, it is determined that a portion of the data in the database is covered by a corresponding dataset pair, satisfying a coverage criterion. For example, Figure 7 The coverage estimator 706 receives a dissimilarity signal 716 (including verified and dissimilarity enhancement pairs 710) and directory information 718. According to an embodiment, the coverage estimator 706 receives the directory information 718 based on the enhancement pairs included in the dissimilarity signal 716 (or otherwise corresponding to the dissimilarity signal 716). For example, according to an embodiment, the coverage estimator 706 compares the embeddings of the enhancement pairs with the embeddings of data in the data directory 418 to obtain a subset of directory information (e.g., directory information 718 in this example) that is semantically similar to the embeddings of the enhancement pairs. According to another embodiment, the coverage estimator 706 receives predicted directory information (e.g., predicted directory information 208) used to generate enhancement pairs as directory information 718. In another example of this embodiment, the coverage estimator 706 determines the semantically similar portions of the predicted directory information to the enhancement pairs to determine directory information 718.

[0102] Continuing with step 1102, the coverage evaluator 706 determines whether coverage of a portion of the data in the database 114 corresponding to the directory information 718 is satisfied by existing dataset pairs of the synthetic data 144. According to an embodiment, the coverage criterion is a threshold that a similarity measure must satisfy between dataset pairs (e.g., dataset pairs 714 of the synthetic data 144 and / or other dataset pairs) and a portion of the data in the database 114 corresponding to the directory information 718. According to an embodiment, the coverage evaluator 706 measures the similarity between the text of the dataset pairs and the text of the directory information 718. According to another embodiment, the coverage evaluator 706 measures the number of dataset pairs that reference the data corresponding to the directory information 718. According to another embodiment, the coverage evaluator 706 measures the similarity between the embeddings of the dataset pairs and the embeddings of the directory information 718. According to an embodiment, the coverage criterion is satisfied if (e.g., at least) a certain percentage of the directory information 718 is covered by dataset pairs. According to another embodiment, the coverage criterion is satisfied if a predetermined number of dataset pairs are semantically similar to the directory information 718.

[0103] In some embodiments, coverage evaluator 706 receives dataset pairs from synthetic data 144. In other embodiments, dataset pairs are included in dissimilarity signals 716 (e.g., dataset pair 714). According to an embodiment, coverage evaluator 706 determines whether a dataset pair covers data corresponding to catalog information 718 in groups (e.g., fragments of dataset pairs, all dataset pairs at once, etc.). According to another embodiment, coverage evaluator 706 evaluates dataset pairs one at a time to see if they cover a portion (or all) of catalog information 718 until enhancement pairs are filtered out from enhancement pairs 710 (e.g., because sufficient catalog information 718 is covered) or all dataset pairs have been evaluated for coverage of catalog information 718. According to another embodiment, coverage evaluator 706 uses catalog information 718 (e.g., embeddings describing its context) as an index to search in synthetic data 144 for one or more dataset pairs that have similarity to catalog information 718 that meets coverage criteria.

[0104] In an embodiment, if the coverage evaluator 706 determines that the coverage criterion is met (i.e., (e.g., sufficient) catalog information 718 corresponding to the enhancement pair has been covered by existing (multiple) dataset pairs), then the coverage evaluator 706 filters enhancement pairs from the enhancement pairs 710 and evaluates the next enhancement pair. For example, as Figure 7As shown, the coverage evaluator 706 provides the query parser 702 with an indication 726 indicating that the enhancement pair has been filtered (or will be filtered), and the query parser 702 parses the next enhancement query. In an alternative embodiment (e.g., where the synthetic data post-processor 204 does not utilize the query parser), the indication 726 is provided to the similarity evaluator 704, and the similarity evaluator 704 evaluates the next enhancement query. In another alternative embodiment (e.g., where the synthetic data post-processor 204 does not utilize the query parser or the similarity evaluator, or where the query is parsed and / or the similarity of the query is evaluated after the coverage evaluation), the coverage evaluator 706 evaluates the next enhancement query. By filtering enhancement pairs that have already covered a portion of the data in the database (in a satisfactory amount or manner) by existing dataset pairs, the synthetic data generator 130 of the coverage evaluator 706 generates different synthetic data for the database. Furthermore, by reducing the overlap in the coverage of dataset pairs to the database 114, embodiments of the coverage evaluator 706 reduce the storage space consumed by the synthetic data 144.

[0105] In one embodiment, if the coverage evaluator 706 determines that the coverage criterion is not met, the coverage evaluator 706 includes the enhancement pair in the filtered pair 708 and processes the next enhancement pair. According to an alternative embodiment, further post-processing occurs after the coverage evaluation, whereby the coverage evaluator provides a new coverage signal to another component of the synthetic data post-processor 204. Figure 7 (Not shown in the image). In such an example, the new coverage signal includes enhancement pairs and / or indications that the coverage of database 114 for enhancement pairs has not yet been satisfied by existing dataset pairs. If the last enhancement pair has been evaluated, the process continues to... Figure 8 Step 806 of flowchart 800. Alternatively, if the last enhancement pair has been evaluated, the next pair in pair queue 402 is provided to cue word generator 202 to generate cue words for generative AI model 132, as described elsewhere in this document.

[0106] Therefore, it has already been about Figure 7-11 An example embodiment of a process for filtering enhancement pairs from a set of enhancement pairs is described. In some embodiments, a synthetic data post-processor 204 (and / or its components) iterates through each enhancement pair until each pair has been filtered out of the enhancement pairs (in which case, no synthetic data is generated) or each pair has been evaluated. Figure 7 As shown, the remaining enhancement pairs are included in the filtered pair 708, and the synthetic data post-processor 204 generates synthetic data including the filtered pair 708 (e.g., as per the description of...). Figure 8 (as described in step 806 of flowchart 800).

[0107] According to one or more embodiments, Figure 7System 700 is configured to iteratively generate synthetic data. For example, synthetic data post-processor 204 according to an embodiment is configured to generate additional enhancement pairs from previously generated enhancement pairs. For example, as Figure 7 As shown and indicated by dashed line 728, the synthetic data post-processor 204 according to the embodiment adds verified enhancement pairs (e.g., filtered pairs 708) to the pair queue 402. In this case, additional enhancement pairs are generated from the verified enhancement pairs added to the pair queue 402. To better understand the operation of the system 700 that iteratively generates synthetic data, regarding... Figure 12 Further description Figure 7 . Figure 12 A flowchart 1200 is shown, illustrating a process for iteratively generating synthetic data according to another example embodiment. According to the embodiment, system 700 operates according to flowchart 1200. Note that not all steps of flowchart 1200 need to be performed in all embodiments. Based on Figure 12 Compared to Figure 7 The following description, other structural and operational embodiments will be apparent to those skilled in the art.

[0108] Flowchart 1200 begins at step 1202. In step 1202, a second cue word is generated using the first enhancement pair and the second prediction directory information, causing the LLM to generate a variant of the first enhancement pair. For example, suppose the filtered pair 708 is added to pair queue 402 (e.g., as...). Figure 7 (As shown in line 728). In this example, queue 402 provides enhancement pair 730 to cue word generator 202, and cue word generator 202 generates new cue words from enhancement pair 730 and prediction directory information. According to an embodiment, cue word generator 202 generates new cue words from enhancement pair 730 and the same prediction directory information used to generate cue words so that generative AI model 132 generates enhancement pair 730 (e.g., Figure 2 (Predicted catalog information 208). According to another embodiment, Figure 4 The directory predictor 406 determines the predicted directory information for the augmented pair 730. In an example of this embodiment, the directory predictor 406 determines the predicted directory information by matching the text of the augmented pair 730 with portions of the data directory 418. In an alternative example of this embodiment, the augmented pair 730 is provided to the embedding model 128 so that the embedding model 128 generates pair embeddings for the augmented pair 730 (e.g., regarding...). Figure 4 and 5 The description method), and the directory predictor 406 determines the enhanced predicted directory information for 730 based on the database embedding and the embedding (e.g., regarding the...). Figure 4 and 6 (Description method).

[0109] In step 1204, in response to the second prompt word being provided to the LLM, a second enhancement pair is received, comprising a second enhanced natural language query and a second enhanced query language query, wherein the second enhanced natural language query is a variant of the first enhanced natural language query, and the second enhanced query language query is a variant of the first enhanced query language query. For example, Figure 7 The synthetic data post-processor 204 receives from the generative AI model 132 one or more additional enhancement pairs (“additional enhancement pairs”) generated based on the cue words including enhancement pair 730. In this case, the additional enhancement pairs include an enhanced NL query as a variant of the enhanced NL query of enhancement pair 730 and an enhanced QL query as a variant of the enhanced QL query of enhancement pair 730.

[0110] In step 1206, synthetic data comprising a first enhancement pair and a second enhancement pair is generated. For example, synthetic data post-processor 204 generates synthetic data 214 comprising enhancement pair 730 (and other enhancement pairs from the initial iteration of synthetic data generation) and enhancement pairs received in step 1206. According to an embodiment, synthetic data post-processor 204 stores synthetic data 214 in batches as synthetic data 144 (e.g., in a first transfer, a first set of enhancement pairs after the generation of filtered pair 708 is stored in storage device 116, in a second transfer, a second set of enhancement pairs generated based on enhancement pair 730 is stored in storage device 116, etc.). According to an alternative embodiment, synthetic data post-processor 204 collects synthetic data for a synthetic data generation “session” and stores the synthetic data as a single transfer in storage device 116. For example, according to an embodiment, synthetic data post-processor 204 iterates through each enhancement pair and subsequent enhancement pairs, and once the iteration is complete and a final set of synthetic data is generated from the enhancement pairs, the synthetic data is transferred to storage device 116 for its storage (e.g., by including the synthetic data in synthetic data 144). According to an embodiment, the synthetic data post-processor 204 (or a component thereof) filters enhancement pairs generated from cue words including enhancement pairs 730 (e.g., in a manner similar to those about...). Figure 8-11 (The way the process is described).

[0111] Therefore, regarding Figure 7 System 700 and Figure 12 Flowchart 1200 illustrates an example of iteratively generating synthetic data. In this example, the synthetic data generator 130 generates additional enhancement pairs from previously generated enhancement pairs until a limit is reached on the number of pairs generated, a limit is reached on the number of "branches" from the enhancement pairs (e.g., the number of subsequent enhancement pairs derived from the enhancement pairs), and no more enhancement pairs are generated that are not filtered by the synthetic data post-processor 204, etc. In this case, Figure 7The synthetic data generator 130 generates a "tree" of augmentation pairs, where the first augmentation pair branches from the first dataset pair, and subsequent augmentation pairs further branch from the first augmentation pair. (About...) Figure 15 An example tree for the augmented pair is described.

[0112] As described in this article, in some examples, the synthetic data post-processor includes a coverage evaluator. For example, as... Figure 7 As shown, the synthetic data post-processor 204 includes a coverage evaluator 706, as per [reference to...]. Figure 7 and 11 As described. In some embodiments, the coverage evaluator 706 is configured to cause the synthetic data generator 130 to repeat the generation of synthetic data. For example, Figure 13 A block diagram of a system 1300 for iteratively generating synthetic data according to an example embodiment is shown. Figure 13 As shown, system 1300 includes storage device 116 (including data directory 418), generative AI model 132, prompter 202, and coverage evaluator 706, as in Figure 1 , 2 The descriptions in sections 4 and 7, as well as elsewhere in this document, are provided. For a better understanding of the operation of System 1300, please refer to... Figure 14 describe Figure 13 . Figure 14 A flowchart 1400 is shown illustrating a process for iteratively generating synthetic data according to an example embodiment. According to the embodiment, system 1300 operates according to flowchart 1400. Not all steps of flowchart 1400 need to be performed in all embodiments. Based on the following... Figure 13 and 14 Other structural and operational embodiments will be apparent to those skilled in the art as described herein.

[0113] Flowchart 1400 begins at step 1402. In step 1402, it is determined that the coverage of a portion of the data in the database does not meet the coverage criteria. For example, such as... Figure 13 As shown, the coverage evaluator 706 receives a validity signal 712 indicating that the enhancement pair is valid and a dataset pair 714. In this example, the coverage evaluator 706 determines that the coverage of a portion of the data in the dataset pair 714 and the enhancement pair by the dataset pair does not meet the coverage criteria. In this example, the coverage evaluator 706 determines that the coverage criteria are not met based on the percentage of data in the database 114 covered by these pairs, the number of pairs associated with that data portion (e.g., the number of pairs of embeddings that satisfy the data similarity criterion with similarity between the embeddings of that data portion), and / or any other criteria suitable for determining that the coverage of these pairs of data is unsatisfactory or that additional pairs should be generated for the corresponding portion(s). Figure 13As shown, the coverage evaluator 706 sends a low coverage signal 1302 to the cue word generator 202, and the process continues to step 1404. According to an embodiment, the low coverage signal 1302 includes an indication of data portions that do not meet the coverage criteria, enhancement pairs, dataset pairs 714, and / or prediction catalog information of data portions.

[0114] In step 1404, a second enhancement pair is generated using LLM based on at least one of the first dataset pair or the first enhancement pair. For example, Figure 13 The cue word generator 202 generates cue word 1304 based on (or otherwise in response to) the low coverage signal 1302 and provides cue word 1304 to the generative AI model 132. In the example, cue word 1304 includes prediction catalog information for the data portion for which pairs are to be generated, previous pairs generated by the generative AI model 132, other dataset pairs (e.g., pairs in queue 402, pairs for data 142), and / or other information suitable for providing to the generative AI model 132 to enable the generative AI model 132 to generate augmented pairs, as described elsewhere herein. Also, Figure 13 As shown, generative AI model 132 generates a second enhancement pair 1306 (referred to herein as "enhancement pair 1306") based on the pair included in cue word 1304. For example, in the example, generative AI model 132 generates enhancement pair 1306 based on one or more of the following: enhancement pairs of enhancement pair 710, dataset pairs of dataset pair 408, and / or any other pairs of NL queries and QL queries described herein. In an alternative embodiment, generative AI model 132 generates multiple enhancement pairs (e.g., including enhancement pair 1306) in response to cue word 1304.

[0115] An example embodiment of the synthetic data generator 130 has been described as "tree-based" synthetic data generation. In this case, the synthetic data generator 130 generates branches of augmented pairs from dataset pairs to increase the coverage of synthetic data 144 over the data in database 114. In this case, the quality of queries generated by generative AI model 132 (e.g., or by language transformation engine 146 utilizing generative AI model 132) is improved through efficient alignment of synthetic data and queries. To better understand tree-based synthetic data generation, this paper describes... Figure 15 . Figure 15 An example tree 1500 (hereinafter referred to as "tree 1500") with an enhancement pair according to an embodiment is shown. Other structural and operational examples will be apparent to those skilled in the art based on the following description. Tree 1500 regarding... Figure 4 The system 400 is described as follows.

[0116] Tree 1500, as shown in the figure, details how to generate reinforcing pairs through iterative pair generation (e.g., as about...). Figure 12 (As described in flowchart 1200). In the example, enhancement pairs are generated based on dataset pairs or previously generated enhancement pairs. For example, tree 1500 shows enhancement pairs 1504A-1504C generated from dataset pair 1502. Dataset pair 1502 is... Figure 4 The dataset is an example of 412. According to an embodiment, Figure 4 The cue word generator 202 generates cue words (e.g., cue word 210) that include dataset pair 412, enabling generative AI model 132 to generate augmentation pairs 1504A-1504C as variations of dataset pair 412. In this case, synthetic data post-processor 204 post-processes augmentation pairs 1504A-1504C to determine whether they should be filtered out from the generated synthetic data. For example, suppose synthetic data post-processor 204 filters out augmentation pair 1504B from the augmentation pairs in any of the ways described herein. In this case, synthetic data generator 130 includes augmentation pairs 1504A and 1504C in pair queue 402. This process is repeated, with cue word generator 202 generating cue words from the next pair in pair queue 402 (e.g., until synthetic data generator 130 (or its sub-components) determines that no more augmentation pairs will be generated from that branch of dataset pair 1502).

[0117] like Figure 15 As shown, enhancement pairs 1506A and 1506B are generated from enhancement pair 1504A, enhancement pairs 1510A and 1510B are generated from enhancement pair 1504C, enhancement pairs 1508A and 1508B are generated from enhancement pair 1506A, enhancement pairs 1512A and 1512B are generated from enhancement pair 1510A, and enhancement pairs 1514A-1514D are generated from enhancement pair 1510A. According to an example embodiment, enhancement pairs 1504B, 1506B, 1508A, 1508B, 1510B, 1512B, 1514A, 1514B, 1514C, and / or 1514D are filtered out from the enhancement pairs, and the synthetic data generator 130 generates synthetic data 214 from the filtered enhancement pairs (e.g., enhancement pairs 1504A, 1504C, 1506A, 1510A, 1512A). According to an alternative embodiment, one or more of enhancement pairs 1504B, 1506B, 1508A, 1508B, 1510B, 1512B, 1514A, 1514B, 1514C, and / or 1514D are included in the filtered enhancement pairs used to generate synthetic data 214. For example, the synthetic data post-processor 204 (or a component thereof) determines in an example that the enhancement pair satisfies a criterion (e.g., a coverage criterion) such that the enhancement pair will be included in the synthetic data 214, but no further enhancement pairs will be generated from the enhancement pair.

[0118] IV. Examples of Synthetic Data Generation Based on Feedback Examples of synthetic data generation have already been described regarding the "tree-based" aspect and similar aspects of generating synthetic data based on dataset pairs. However, the embodiments described herein are not limited thereto. For example, some embodiments of the synthetic data generator 130 generate synthetic data based on user feedback (e.g., customer feedback). To better understand such embodiments, this document describes... Figure 16 . Figure 16 A block diagram of a system 1600 for generating synthetic data according to an example embodiment is shown. Figure 16 As shown, system 1600 includes, as per... Figure 1 The described synthetic data generator 130 and storage device 116 (storing synthetic data 144), as well as synthetic data storage criteria 1606, are described. According to an embodiment, the synthetic data storage criteria are a set of rules that the synthetic data generated by the synthetic data generator 130 must follow. Regarding... Figure 23 and 24 And other parts of this document describe additional details regarding synthetic data storage guideline 1606. Also, Figure 16 As shown, the synthetic data generator 130 includes a corrector 1602 (also referred to as a "correction system" or "correction component") and a synthetic data post-processor 1604, each of which is implemented as a sub-service / component of the synthetic data generator 130. According to an embodiment, the synthetic data generator includes, as described above... Figure 2 The described prompt word generator 202 and / or as about Figure 4 The described pair includes queue 402, embedding model 404, and / or catalog predictor 406. According to an embodiment, synthetic data post-processor 1604 is another example of synthetic data post-processor 204. Alternatively, synthetic data post-processor 1604 is a post-processing circuit separate from synthetic data post-processor 204.

[0119] To better understand the operation of System 1600, regarding... Figure 17 describe Figure 16 . Figure 17 A flowchart 1700 is shown illustrating a process for generating synthetic data according to an example embodiment. According to the embodiment, system 1600 operates according to flowchart 1700. Note that not all steps of flowchart 1700 need to be performed in all embodiments. Based on Figure 16 and 17 The following description, other structural and operational embodiments will be apparent to those skilled in the art.

[0120] Flowchart 1700 begins at step 1702. In step 1702, an indication of negative feedback for a first query language query generated by the LLM based on a first natural language query is received. For example, indication 1608 is received for corrector 1602. According to an embodiment, indication 1608 is an indication of negative feedback for a QL query generated by the generative AI model 132 based on an NL query. According to an embodiment, personal identification information is removed from indication 1608. According to an embodiment, corrector 1602 receives indications of negative and / or positive feedback for multiple QL queries at once.

[0121] In step 1704, a correction pair is generated based on the instruction and the first natural language query. The correction pair includes the first natural language query and a corrected query language query, where the corrected query language query is a syntactically valid transformation of the first natural language query. For example, corrector 1602 generates correction pair 1610 based on instruction 1608 and an NL query corresponding to instruction 1608. Correction pair 1610 includes the NL query and a corrected QL query. The corrected QL query is a syntactically valid transformation of the NL query. According to embodiments, and as per the relevant provisions... Figure 19 (And elsewhere in this document) the corrector 1602 utilizes telemetry data to generate the correction pair 1610. According to another embodiment, and as described regarding... Figure 20 As described (and elsewhere herein), the corrector 1602 utilizes a generative AI model (e.g., generative AI model 132) to generate the correction pair 1610. According to embodiments, the corrector 1602 (or another component of the synthetic data generator 130, such as a query parser)... Figure 16 (Not shown) The QL query for the correction pair 1610 is parsed to determine whether the QL query is valid. In some embodiments, the corrector 1602 iteratively attempts to correct the pair to generate the correction pair 1610.

[0122] In step 1706, it is determined that the calibration pair meets the criteria for synthetic data storage. For example, Figure 16 The synthetic data post-processor 1604 determines whether the correction pair 1610 meets the synthetic data storage criterion 1606. Examples of the synthetic data storage criterion 1606 include, but are not limited to, coverage criteria suitable for determining whether the coverage of the data of the correction pair 1610 on the database 114 overlaps with the coverage of the existing synthetic data on the data of the database 114 (e.g., similar to the coverage of the data of the database 114 by the correction pair 1610). Figure 11 The description method), suitable for determining whether the correction pair 1610 is (e.g., too) similar to existing synthetic data, is based on similarity criteria (e.g., similar to the similarity to...). Figure 10 The description method), suitable for determining the consistency criteria of the expected transformation of the corrected QL query of the corrected pair 1610 with the expected transformation of the NL query of the corrected pair 1610 (e.g., regarding the... Figure 23and 24 (and other methods further described elsewhere in this document), and / or any other criteria suitable for determining that correction pair 1610 is a suitable dataset pair to be included as synthetic data 144.

[0123] In step 1708, the correction pair is stored as synthetic data in the synthetic data storage. For example, Figure 16 The post-processor 1604 stores the correction pair 1610 as part of the synthesized data 144 in the storage device 116. For example, as Figure 16 As shown, the synthetic data post-processor 1604 sends a storage signal 1612 (including a correction pair 1610) to the storage device 116 so that the correction pair 1610 is stored as synthetic data of synthetic data 144. In this case, the synthetic data generator 130 automatically generates synthetic data for use by the language conversion model 146 in future query generation to cover gaps and / or otherwise improve the quality of query generation.

[0124] This document has already described flowchart 1700 regarding instructions to receive negative feedback. In some embodiments, the synthetic data generator 130 receives instructions to receive positive feedback. Examples of positive feedback include, but are not limited to, instructions from a client that the query generated by the generative AI model 132 is an acceptable transformation of an NL query, instructions from a client to execute a QL query generated by the generative AI model 132 and corresponding to the NL query provided in the prompts for the model, etc. In an example of receiving positive feedback, the synthetic data generator 130 provides instructions to the synthetic data post-processor 1604. In this case, the synthetic post-processor 1604 verifies the syntax of the QL query, verifies the consistency between the NL query and the QL query, evaluates the similarity between the NL query-QL query pair, evaluates the coverage of the database 114 regarding the QL query, and / or otherwise post-processes the QL query and the corresponding NL query that received positive feedback. In this case, if the pair is valid and otherwise meets the criteria of the synthetic data post-processor 1604, the synthetic data generator 130 (or a component thereof) includes the pair in the synthetic data 144. In this case, the synthetic data generator 130 expands the synthetic data 144 based on the successful conversion from NL query to QL query, thereby improving the quality of future query generation by the language conversion engine 146.

[0125] In the example of feedback-based synthetic data generation, the synthetic data generator 130 is configured in various ways to generate synthetic data. For example, Figure 18 A block diagram of a system 1800 for generating synthetic data according to another example embodiment is shown. Figure 18 As shown, system 1800 includes, as about Figure 1The described applications 120, generative AI models 132, and synthetic data 144, as well as regarding... Figure 16 The described synthetic data generator 130 (including a corrector 1602 and a synthetic data post-processor 1604) and synthetic data storage criteria 1606, as well as session telemetry 1818, are described. Session telemetry 1818 includes... Figure 1 The database 114, database engine 134, language conversion engine 146, and other components of system 100, as well as any sub-components described elsewhere herein (e.g., data and usage measurements related to the use of applications such as application 120 and / or by users utilizing applications such as application 120). Also, Figure 18 As shown, the synthetic data generator 130 includes a feedback preprocessor 1802, a feedback indicator 1804, and a syntax validator 1806, each of which is a sub-service / component of the synthetic data generator 130. Furthermore, Figure 18 The corrector 1602 includes a telemetry evaluator 1808 and a self-correcting agent 1810 (including a competence verifier 1812, a prompter 1814, and a grammar evaluator 1816), each of which is a sub-service / component of the corrector 1602. In one or more alternative examples, the competence verifier 1812, the prompter 1814, and the grammar evaluator 1816 are implemented separately from the self-correcting agent 1810 (e.g., as separate components of the corrector 1602, the synthetic data generator 130, and / or the system 1800). For example, in an alternative example, the grammar evaluator 1806 performs grammar evaluation on the self-correcting agent 1810 (e.g., instead of the grammar evaluator 1816). According to embodiments, the grammar evaluator 1806 and / or the grammar evaluator 1816 are... Figure 7 Further examples of query parser 702.

[0126] Figure 18 The feedback preprocessor 1802 is configured to receive feedback 1820 and preprocess feedback 1820 from application 120. In the example, application 120 responds to the user interface (UI) of application 120. Figure 18Feedback 1820 is generated based on user interaction (not shown) to indicate whether the QL query generated by the language conversion engine 146 is a satisfactory conversion of the NL query. In the example, feedback 1820 includes the QL query generated by the language conversion engine 146 (e.g., the QL query for which feedback 1820 is provided), a binary decision on whether the conversion is satisfactory (e.g., a positive indication indicating its satisfaction or a negative indication indicating its unsatisfaction), a rating indicating the degree of satisfaction or unsatisfaction of the conversion (e.g., a rating from 1 to 5, a rating from 1 to 10, etc.), one or more comments on the user feedback, answers to one or more questions about the generated QL query, and / or any other information associated with the feedback on the generated QL query. According to an embodiment, the user is prompted to provide feedback 1820 via the UI of application 120 (e.g., after the QL query is generated by the language conversion engine 146, after the QL query is executed, after the user modifies the QL query, as part of a customer satisfaction survey, etc.). According to an embodiment, a feedback preprocessor 1802 removes personally identifiable information from the feedback 1820. According to an embodiment, the feedback preprocessor 1802 removes information that is not available to other components of the synthetic data generator 130 (e.g., to reduce data transfer between components). Figure 18 As shown, the feedback preprocessor 1802 processes the feedback 1820 into processed feedback 1822.

[0127] Feedback indicator 1804 is configured to analyze processed feedback 1822 and determine whether the processed feedback 1822 indicates positive feedback (e.g., the user is satisfied with the generated QL query) or negative feedback (e.g., the user is dissatisfied with the generated QL query). In the example, feedback indicator 1804 determines whether processed feedback 1822 indicates positive or negative feedback based on binary indications (e.g., "thumbs up" or "thumbs down", "yes" or "no", etc.), ratings that satisfy (or do not satisfy) a positive or negative feedback threshold (e.g., in a non-limiting example, a rating of 1 or 2 on a 1-5 scale indicates negative feedback), long-form responses indicating user satisfaction or dissatisfaction with the QL query, and / or any other type of feedback suitable for analysis to determine whether processed feedback 1822 indicates user (or application) satisfaction with the generated QL query. Figure 18 As shown, if feedback indicator 1804 determines that the processed feedback 1822 is positive feedback, then feedback indicator 1804 generates an indication of positive feedback 1824 ("indication 1824" herein). Also as... Figure 18 As shown, if feedback indicator 1804 determines that the processed feedback 1822 is negative feedback, then feedback indicator 1804 generates an indication of negative feedback 1830 ("indication 1830" herein). Indication 1830 is as follows regarding... Figure 16Another example of the described instruction 1608.

[0128] In the example, Figure 18 The synthetic data generator 130 operates in various ways. For example, as... Figure 18 As shown, if feedback indicator 1804 generates indicator 1824, indicator 1824 is provided to syntax validator 1806. Syntax validator 1806 verifies the QL query corresponding to indicator 1824 (i.e., the QL query generated by language conversion engine 146). If the QL query is valid, syntax validator 1806 provides verified query signal 1826 to synthetic data post-processor 1604. In this case, synthetic data post-processor 1604 post-processes the verified QL query and the NL query from which the verified QL query was generated (e.g., in the prompt words provided to generative AI model 132 that cause generative AI model 132 to generate the verified QL query, as described elsewhere herein) to determine whether to include the pair in synthetic data 144. If the synthetic data post-processor 1604 determines that the pair should be included in the synthetic data 144 (e.g., by determining that the pair satisfies the synthetic data storage criterion 1606), then the synthetic data post-processor 1604 stores the pair in the synthetic data 144 via a storage signal 1846 including the pair. If the syntax validator 1806 determines that the QL query is invalid, then the syntax validator 1806 provides an invalid query signal 1828 to the self-correcting agent 1810. In this case, the self-correcting agent 1810 attempts to do so in a manner described elsewhere herein, particularly regarding... Figure 20 and 21 This generates a correction pair that includes the correction of the QL query.

[0129] If feedback indicator 1804 generates indication 1830, then indication 1830 is provided to corrector 1602. (As per...) Figure 16 and Figure 17 As discussed in step 1704 of flowchart 1700, in this example, the corrector 1602 is configured to generate correction pairs in various ways (e.g., Figure 16 The calibration pair 1610). For example, in some examples, the calibration pair is generated for the corrector 1602 based on session telemetry 1818. To better illustrate this example operation of system 1800, regarding... Figure 19 describe Figure 18 . Figure 19 A flowchart 1900 is shown illustrating a process for generating a correction pair according to an example embodiment. According to the embodiment, system 1800 operates according to flowchart 1900. Note that not all steps of flowchart 1900 need to be performed in all embodiments. Based on Figure 19 Compared to Figure 18The following description, other structural and operational embodiments will be apparent to those skilled in the art.

[0130] Flowchart 1900 begins at step 1902. In step 1902, a second query language query performed against the database is identified. For example, telemetry evaluator 1808 receives indication 1830 and session telemetry 1818. According to an embodiment, telemetry evaluator 1808 (e.g., selectively) receives session telemetry 1818 corresponding to a time frame (e.g., minutes, hours, days, etc.) or session that generated the QL query corresponding to indication 1830. For example, according to an embodiment, feedback 1820 includes a session identifier (ID) of the session in which the QL query was generated. In this example embodiment, telemetry evaluator 1808 obtains a portion of session telemetry 1818 corresponding to the session ID. In this example, telemetry evaluator 1808 identifies one or more queries performed against database 114 during the session based on session telemetry 1818, and the flow continues to step 1904. If the telemetry evaluator 1808 fails to identify the executed query based on session telemetry 1818, the telemetry evaluator 1808 provides instruction 1834 to the self-calibrating agent 1810. In this case, instruction 1834 is used to indicate any information included in instruction 1830 (e.g., a QL query receiving negative feedback), and the self-calibrating agent 1810 attempts to generate a calibration pair. Regarding Figure 20 and 21 Additional details regarding the operation of the self-calibrating agent 1810 are described elsewhere in this document.

[0131] In step 1904, it is determined that the second query language query corresponds to the first natural language query. For example, telemetry evaluator 1808 determines that the QL query identified in step 1902 corresponds to the NL query associated with feedback 1820 (i.e., the NL query from which language conversion engine 146 generates a QL query with negative feedback (also referred to herein as an "incorrect QL query"). In this example, telemetry evaluator 1808 determines that the QL query identified in step 1902 corresponds to an NL query based on: the temporal proximity of executing the identified QL query and generating the incorrect QL query, a similarity measure in the text of the NL query and the identified QL query, a similarity measure in the embedding of the NL query and the embedding of the identified QL query, and / or any other way of determining that the QL query corresponds to the NL query. If it is determined that the QL query corresponds to the NL query, the process continues to step 1906. If no identified QL query is determined to correspond to an NL query, telemetry evaluator 1808 provides indication 1834 to self-correction 1810. In this case, instruction 1834 is used to indicate any information included in instruction 1830, and self-correcting agent 1810 attempts to generate a correction pair, as described elsewhere in this document.

[0132] In step 1906, a calibration pair is generated based on a first natural language query and a second query language query. For example, telemetry estimator 1808 generates calibration pair 1832 based on the NL query and the QL query determined in step 1904 to correspond to the NL query. In some embodiments, telemetry estimator 1808 determines in step 1904 that multiple QL queries (e.g., potentially) correspond to the NL query. In this case, telemetry estimator 1808 generates multiple calibration pairs according to the embodiment, each calibration pair being based on the NL query and the corresponding QL query among the multiple QL queries. Alternatively, telemetry estimator 1808 selects the QL query that has the highest similarity to the NL query (e.g., based on embedded text matching or semantic matching) and generates calibration pair 1832 based on that QL query and the NL query.

[0133] like Figure 18 As shown, the telemetry evaluator 1808 provides a correction pair 1832 to the synthetic data post-processor 1604. The synthetic data post-processor 1604 determines whether the correction pair 1832 meets the synthetic data storage criterion 1606 (e.g., regarding...). Figure 17 The process involves step 1706 of flowchart 1700 (or as described elsewhere herein), and if so, storing the correction pair 1832 in the synthetic data 144 via storage signal 1846. In embodiments where the telemetry evaluator 1808 generates multiple correction pairs, the synthetic data post-processor 1604 filters the multiple correction pairs based on synthetic data storage criteria 1606 and stores (e.g., one or more) of the filtered correction pairs in the synthetic data 144 via storage signal 1846.

[0134] Such as about Figure 18 As described, the corrector 1602 generates correction pairs in various ways. For example, the self-calibrating agent 1810 operates in a manner that generates correction pairs. To better understand the operation of the self-calibrating agent 1810, regarding... Figure 20 Further description Figure 18 . Figure 20 A flowchart 2000 of a process for generating a correction pair according to another example embodiment is shown. According to the embodiment, system 1800 operates according to flowchart 2000. Note that flowchart 2000 does not need to be executed in all embodiments. Figure 20 Compared to Figure 18 The following description, other structural and operational embodiments will be apparent to those skilled in the art.

[0135] In some embodiments, and as Figure 18As shown, the self-correcting agent 1810 includes a suitability verifier 1812. The suitability verifier 1812 determines whether an NL query is suitably converted into a QL query (e.g., based on synthetic data storage criterion 1606 or other criteria). Regarding... Figure 22 Additional details about the eligibility verifier 1812 are described elsewhere in this document. If an NL query eligibility transition occurs, the eligibility verifier 1812 provides an eligibility query signal 1836 to the prompter 1814, and the process proceeds to flowchart 2000 as further described herein.

[0136] Flowchart 2000 includes step 2002. In step 2002, the LLM is prompted to generate a correction pair based on the instruction and the first natural language query. For example, Figure 18 The prompter 1814 generates a prompt word 1838 and provides this prompt word to the generative AI model 132, enabling the generative AI model 132 to generate a correction pair 1840 based on indication 1830 (and / or indication 1834 and / or invalid query signal 1828) and an NL query, wherein the correction pair 1840 includes the NL query and a corrected QL query generated by the generative AI model 132. For example, according to an embodiment, the prompt word 1838 indicates to the generative AI model 132 that an incorrect QL query is an unsatisfactory transformation of the NL query. According to embodiments that provide feedback other than binary feedback (or feedback in addition to binary feedback) for incorrect QL queries, the prompt word 1838 includes this feedback. For example, in such an example, the prompt word 1838 includes user comments, user ratings, etc. In this way, the prompt word 1838 provides the generative AI model 132 with additional context to consider when generating the correction pair 1840.

[0137] like Figure 18 As shown, the syntax evaluator 1816 receives the correction pair 1840 and determines whether the corrected QL query of the correction pair 1840 is syntactically valid. If so, the syntax evaluator 1816 provides the verified correction pair 1842 to the synthetic data post-processor 1604 (i.e., in this case, the correction pair 1840 is verified afterward). The synthetic data post-processor 1604 determines whether the verified correction pair 1842 meets the synthetic data storage criterion 1606, and if so, stores the verified correction pair 1842 in the synthetic data 144 via the storage signal 1846.

[0138] If the grammar evaluator 1816 determines that the corrected QL query is not syntactically valid, the self-correcting agent 1810 operates in various ways to attempt to regenerate a valid corrected QL query. To better understand this iterative attempt to generate a valid corrected QL query, regarding... Figure 21 Further description Figure 18 . Figure 21 A flowchart 2100 is shown illustrating a process for iteratively generating correction pairs according to an example embodiment. According to the embodiment, system 1800 operates according to flowchart 2100. Note that not all steps of flowchart 2100 need to be performed in all embodiments. Based on Figure 21 Compared to Figure 18 The following description, other structural and operational embodiments will be apparent to those skilled in the art.

[0139] Flowchart 2100 begins at step 2102. In step 2102, first prompt words are generated that enable the LLM to generate candidate pairs based on the indication and the first natural language query. For example, suppose... Figure 18 The prompter 1814 generates a prompt word 1838 and provides the prompt word 1838 to the generative AI model 132 in a manner similar to that described with respect to flowchart 2000. In this case, the generative AI model 132 provides a correction pair 1840 (referred to as "candidate pair 1840" in this example) to the grammar evaluator 1816, and the process continues to step 2104.

[0140] In step 2104, candidate pairs are determined to be syntactically invalid. For example, suppose... Figure 18 The syntax evaluator 1816 determines that candidate pair 1840 is syntactically invalid. According to an embodiment, if the syntax evaluator 1816 cannot successfully parse the corrected QL query (e.g., based on rules of the query language that generated the QL query), then candidate pair 1840 is determined to be syntactically invalid. Figure 18 As shown, if candidate pair 1840 is syntactically invalid, the syntactic evaluator 1816 provides a retry signal 1844 to the prompter 1814, and the process continues to step 2106.

[0141] In step 2106, a second cue word is generated so that the LLM can generate a correction pair based on the indication, the first natural language query, and the determination that the candidate pair is grammatically invalid. For example, Figure 18 The prompter 1814 generates a second prompt word ( Figure 18 (not shown in the image), and provides a second prompt word to the generative AI model 132, so that the generative AI model 132 generates another correction pair based on the indication (e.g., indication 1830, indication 1834 and / or invalid query signal 1828), NL query and retry signal 1844. Figure 18 (Not shown in the image).

[0142] In step 2108, it is determined that the correction pair is grammatically valid. For example, Figure 18The grammar evaluator 1816 receives another correction pair generated by the generative AI model 132 in response to the second cue word generated in step 2106, and determines whether the other correction pair is grammatically valid. If the other correction pair is grammatically valid, the grammar evaluator 1816 provides the other correction pair as a valid correction pair 1842 to the synthetic data post-processor 1604. In this case, the synthetic data post-processor 1604 determines whether the valid correction pair 1842 meets the synthetic data storage criterion 1606, and if so, stores the valid correction pair 1842 in the synthetic data 144 via the storage signal 1846.

[0143] If another correction pair is syntactically invalid, the grammar evaluator 1816 provides another retry signal to the prompter 1814 and repeats the process of flowchart 2100. In some embodiments, each successive prompt word generated by the prompter 1814 includes an indication that the previously generated candidate QL query is an invalid query (e.g., any, some, all), an NL query, and information associated with that indication (e.g., indication 1830, indication 1834, and / or invalid query signal 1828). According to an embodiment, if no valid correction pair is generated after multiple attempts, the self-correcting agent 1810 terminates the iterative attempts to generate a valid correction pair. In the example, the number of attempts is a predetermined number (e.g., set by the developer of the self-calibrating agent 1810, the administrator associated with the self-calibrating agent 1810, organizational policies associated with the self-calibrating agent 1810, etc.), limited based on the backlog of feedback to be evaluated exceeding a threshold, limited based on resource usage limitations (e.g., the number of times the generative AI model 132 can be invoked in a particular session, budget constraints, etc.), limited based on the amount of time the self-calibrating agent 1810 spends attempting to generate a valid calibration pair, and / or determined in any other way suitable for limiting the number of attempts by the iterative self-calibrating agent 1810 to generate a valid calibration pair, as described elsewhere herein and / or as will be otherwise understood by one of ordinary skill in the art(s) that benefit from this disclosure.

[0144] like Figure 18 As shown, the self-correcting agent 1810 includes a eligibility verifier 1812. The eligibility verifier 1812 verifies whether an NL query is eligible to be converted into a QL query. In embodiments, the eligibility verifier 1812 operates in various ways. For example, Figure 22 A flowchart 2200 is shown for a process of determining query eligibility according to another example embodiment. According to the embodiment, eligibility verifier 1812 operates according to flowchart 2200. Note that not all steps of flowchart 2200 need to be performed in all embodiments. Based on the following... Figure 18 and Figure 22Other structural and operational embodiments will be apparent to those skilled in the art as described herein.

[0145] Flowchart 2200 includes step 2202. In step 2202, the eligibility of a first natural language query to be converted is determined based on at least one of the user account's permissions or the availability of tables in the database. For example, eligibility verifier 1812 receives indication 1834 (or indication 1830) and / or invalid query signal 1828, and determines whether the corresponding NL query is eligible to be converted to a QL query based on the user account's permissions and / or the availability of tables in database 114. For example, suppose a user is attempting to generate a QL query to execute against a table in database 114. In this case, eligibility verifier 1812 determines that the user's account has authorization to access data in the tables of database 114 (and / or manipulate data depending on the type of query to be generated). If the user account has authorization, eligibility verifier 1812 generates an eligibility query signal 1836, and the process proceeds as follows: Figure 20 The process continues as described in flowchart 2000. If the user account is not authorized, the eligibility verifier 1812 refuses to generate a calibration pair for the NL query. By determining whether the user account is authorized, access to the data by unauthorized accounts or accounts without authorization is prevented. In this way, the eligibility verifier 1812 prevents unauthorized users from generating synthetic data associated with queries they are not authorized to execute, thereby maintaining the quality of the synthetic data. For example, malicious entities cannot degrade the quality of synthetic data in a portion of database 114 that they would not otherwise be authorized to access.

[0146] In another example, suppose the NL query references a specific table (or another data structure in database 114). In this example, the eligibility verifier 1812 determines whether the NL query is eligible based on whether the table exists in database 114. If the table exists (e.g., and the user account is authorized to access it), the eligibility verifier 1812 generates an eligibility query signal 1836, and the process proceeds regarding... Figure 20 The process described in flowchart 2000 continues. If the table does not exist, the eligibility verifier 1812 prevents the generation of calibration pairs. In this way, if the table (or other structure of database 114 referenced by NL queries) is not available in database 114, the computational resources utilized by the self-calibrating agent 1810 are reduced. Furthermore, if the table (or other structure) is not available, the computational resources utilized by the generative AI model 132 are also reduced, because if the table (or other structure) is not available, the self-calibrating agent 1810 does not prompt the generative AI model to generate calibration pairs.

[0147] As described in this article, Figure 16The synthetic data post-processor 1604 processes the correction pairs generated by the syntax validator 1806 and / or the operation of the corrector 1602 (and / or its sub-components). In the example, the synthetic data post-processor 1604 includes a query parser, a similarity estimator, and / or a coverage estimator, which operate in a similar manner to the query parser 702, the similarity estimator 704, and the coverage estimator 706, as described above. Figure 7 (And elsewhere in this document) as described. For example, logic including, when executed, parsing the corrected query, evaluating the similarity between the corrected query and existing dataset pairs, and evaluating the coverage of existing dataset pairs and the corrected query. In some embodiments, the synthetic data post-processor 1604 is configured to evaluate the correction pair relative to other synthetic data storage criteria (e.g., in addition to or instead of parsing, evaluating similarity, and / or evaluating coverage). For example, Figure 23 A block diagram of a system 2300 for evaluating the consistency of calibration pairs according to an example embodiment is shown. Figure 23 As shown, system 2300 includes, as about Figure 1 The described generative AI model 132 and such as about Figure 16 The described synthetic data post-processor 1604. Also, Figure 23 As shown, the synthetic data post-processor 1604 includes a consistency verifier 2302 (including a consistency hinter 2304 and a consistency evaluator 2306), which is implemented as a sub-service / component of the synthetic data post-processor 1604. The consistency verifier 2302 evaluates the consistency in the transformation from the NL query of the calibration pair to the calibration QL query of the calibration pair.

[0148] To better understand the operation of System 2300, regarding... Figure 24 describe Figure 23 . Figure 24 A flowchart 2400 is shown illustrating a process for evaluating the consistency of a calibration pair according to an example embodiment. According to the embodiment, a consistency verifier 2302 operates according to flowchart 2400. Note that not all steps of flowchart 2400 need to be performed in all embodiments. Figure 23 and 24 The following description, other structural and operational embodiments will be apparent to those skilled in the art.

[0149] Flowchart 2400 begins at step 2402. In step 2402, prompt words are generated to enable the LLM to generate candidate natural language queries based on the calibrated query language query. In step 2402, consistency prompter 2304 generates prompt words 2310 to enable generative AI model 132 to generate candidate NL queries 2312 based on the calibrated QL query. For example, suppose as follows Figure 23As shown, the consistency prompter 2304 receives a correction pair 2308 (which is another embodiment of correction pair 1610, correction pair 1832, and / or verified correction pair 1842, as described elsewhere herein). In this case, the consistency prompter 2304 generates a prompt term 2310 from the QL query included in the correction pair 2308 (e.g., in some examples, the NL query is not included in the prompt term 2310). According to an embodiment, the prompt term 2310 includes instructions for generating a natural language query from the QL query.

[0150] In step 2404, it is determined that the similarity between the first natural language query and the candidate natural language query satisfies a consistency criterion. For example, consistency evaluator 2306 determines the similarity between the NL query of the correction pair 2308 and the candidate NL query 2312. According to an embodiment, consistency evaluator 2306 measures the similarity between the NL query and the candidate NL query 2312 based on text matching between the two queries. According to another embodiment, consistency evaluator 2306 measures the similarity between the embedding of the NL query and the embedding of the candidate NL query 2312. In this case, the embedding of the NL query of the correction pair 2308 is obtained from language conversion engine 146 (e.g., in an implementation where language conversion engine 146 generates or otherwise obtains the embedding of the NL query), obtained using embedding model 128 (e.g., in a manner similar to that described elsewhere herein), etc. In an implementation, consistency evaluator 2306 obtains the embedding of the candidate NL query 2312 from generative AI model 132 using embedding model 128, etc. Examples of consistency criteria include, but are not limited to, thresholds that must be met for the similarity metric between the NL query and the candidate NL query, thresholds that must be met for the similarity difference between queries, etc. If the consistency evaluator 2306 determines that the similarity between the NL query and the candidate NL query 2312 meets the consistency criteria, then the consistency evaluator 2306 generates a consistency pair signal 2316. Alternatively, the consistency evaluator 2306 stores this pair in the synthetic data 144 (e.g., via...). Figure 18 (Storage signal 1846). According to an embodiment, Figure 23 Other components of the synthetic data post-processor 1604 (not shown) (e.g., parser, similarity estimator, coverage estimator, etc.) receive the consistency pair signal 2316 and further post-process the correction pair 2308, as described elsewhere herein. Figure 24 In the manner described, embodiments of the consistency validator 2302 improve the automatic generation of synthetic data based on negative customer feedback by reverse engineering the corrected QL query to verify that it is a valid context transformation of the NL query. In other words, the consistency validator 2302 determines that the corrected QL query conforms to the expected query language from the NL query.

[0151] About Consistency Validator 2302 Figure 24 Flowchart 2400 is described as verifying the calibration pair (e.g., by...) Figure 16 The consistency verifier described herein is not limited to the consistency verifier generated by the corrector 1602 and / or its sub-components. For example, in an alternative (or additional) embodiment, the consistency verifier 2302 determines the consistency between QL queries for valid query signals 1826. By verifying the consistency between QL queries that receive positive feedback from the user, such an embodiment of the consistency verifier 2302 verifies that the QL query that receives positive feedback is a contextually valid transition of the NL query. For example, if the user incorrectly provides positive feedback, the consistency verifier 2302 determines that the similarity between the NL query and the candidate NL query determined from the QL query does not meet the consistency criteria and prevents the pair from being included in the synthetic data 144.

[0152] Already about Figure 18 System 1800 and especially about Figure 24 Flowchart 2400 describes an embodiment of consistency validator 2302; however, the embodiments of the consistency validator described herein are not limited thereto. For example, according to an alternative (or additional) embodiment, consistency validator 2302 (or another consistency validator of synthetic data generator 130) verifies the augmented pairs generated from the dataset pairs (e.g., Figure 2 Enhancement for 212, Figure 7 The consistency of 710, etc. is enhanced. In this case, the consistency verifier 2302 acts as a consensus verifier. Figure 8 The flowchart 800 describes the iterative filtering process as part of a single-iteration post-processing step as part of (multiple) enhancement pairs and / or as independent of the iterative filtering process described above. Figure 2 , 4 And the post-processing procedures described in 7 are used to operate.

[0153] V. Instances generated for query In the examples, a language conversion engine (such as language conversion engine 146) selects synthetic data generated by synthetic data generator 130 to include in prompts for generative AI model 132, enabling generative AI model 132 to generate QL queries. In these examples, the selected synthetic data provides additional context for the QL queries to be generated through example transformations of natural language input to QL queries. In this way, the embodiments improve the quality of QL queries generated by generative AI model 132 and reduce the possibility of generative AI model 132 experiencing hallucinations during the query generation process.

[0154] Example embodiments of the language conversion engine 146 are configured in various ways to generate QL queries. For example, Figure 25A block diagram of a system 2500 for generating queries according to an example embodiment is shown. Figure 25 As shown, system 2500 includes, as per... Figure 1 The described application 120, generative AI model 132, synthetic data 144, and language conversion engine 146 (including preprocessor 122, prompter 124, and postprocessor 126), as well as query embedding 2508. Also, Figure 25 As shown, the preprocessor 122 includes a pattern processor 2502, a synthetic data filter 2504, and an embedded comparator 2506, each of which is implemented as a component and / or subcomponent of the preprocessor 122. System 2500 is described below.

[0155] The schema processor 2502 receives a request to generate a QL query and determines the schema of the database 114 associated with the request. For example, such as Figure 25 As shown, the pattern processor 2502 receives an NL query 2510 from application 120, where the NL query 2510 is a request to generate a QL query based on the NL input of the NL query 2510. Depending on the implementation, the pattern processor 2502 may base its decisions on information included in the NL query 2510 (e.g., table name, column name, user account identifier, etc.), text analysis of the NL query 2510, and / or the embedding of the NL query 2510 with the context of a specific part of the database 114. Figure 25 The semantic similarity between (not shown in the image) is used to determine the pattern. For example... Figure 25 As shown, the pattern processor 2502 provides a pattern signal 2512 to the synthetic data filter 2504. In an embodiment, the pattern signal 2512 includes the NL query 2510, the pattern determined by the pattern processor 2502, and / or any information related to QL query generation that is provided to the language conversion engine 146 (e.g., from application 120) and / or generated / determined by the pattern processor 2502.

[0156] Synthetic data filter 2504 filters the synthetic data to determine a subset of synthetic data compatible with the pattern determined by pattern processor 2502. For example, such as Figure 25 As shown, the synthetic data filter 2504 receives the pattern signal 2512 from the pattern processor 2502 and receives synthetic data 2514 (e.g., a subset or all of it) from the synthetic data 144 (e.g., from...). Figure 1The storage device 116 receives the data and generates filtered synthetic data 2516. According to one embodiment, the synthetic data filter 2504 selects pairs from the synthetic data 2514 based on the pattern signal 2512 to generate filtered synthetic data 2516. According to another embodiment, the synthetic data filter 2504 removes pairs from the synthetic data 2514 from the consideration based on the pattern signal 2512 to generate filtered synthetic data 2516. According to another embodiment, the synthetic data filter 2504 selectively receives the synthetic data 2514 based on the pattern signal 2512 (e.g., by searching for filtered synthetic data 2516 in the synthetic data 144 using the pattern (or its embedding) of the pattern signal 2512 as an index). According to one embodiment, the filtered synthetic data 2516 includes one or more pairs of patterns in the synthetic data 2514 that are semantically similar to the pattern signal 2512. For example, according to an embodiment, the synthetic data filter 2504 generates filtered synthetic data based on the embedding of the pattern of the pattern signal 2512 and the embedding of the synthetic data 2514 (comparison and / or similarity measurement, etc.). According to an embodiment, the embedding of the pattern of the pattern signal 2512 is obtained by the pattern processor 2502, obtained from the data catalog, and / or obtained from the embedding model (e.g., the synthetic data filter 2504 obtains it using the embedding model 128, or another component or subcomponent of the system 2500). Figure 1 The embedding of the synthetic data 2514 is obtained from the embedding model 128. According to an embodiment, the embedding of the synthetic data 2514 is stored as part of the synthetic data 144 or obtained from the embedding model (e.g., the synthetic data filter 2504 obtains it using the embedding model 128, or another component or subcomponent of the system 2500 obtains it using the embedding model 128). Figure 25 As shown, the synthetic data filter 2504 provides the filtered synthetic data 2516 to the embedded comparator 2506.

[0157] Embedding comparator 2506 determines synthetic data to be included in the prompt words for the generative AI model 132 based on the embeddings of the NL query. For example... Figure 25 As shown, the embedding comparator 2506 receives filtered synthetic data 2516 from the synthetic data filter 2504, receives a query embedding 2508, and generates a preprocessed signal 2518. In an embodiment, the query embedding 2508 describes the semantic context of the NL query 2510. According to an embodiment, the query embedding 2508 is obtained using the embedding model 128. According to an embodiment, the embedding comparator 2506 selects pairs from the filtered synthetic data 2516 that are semantically similar to the NL query 2510. For example, in this example, the embedding comparator 2506 selects pairs based on a similarity metric between the query embedding 2508 and the corresponding embeddings of the pairs. Regarding Figure 27And other parts of this document describe the comparison query embedding 2508 and additional details about the embedding. In the example, the preprocessing signal 2518 includes semantic similarity pairs, NL query 2510, and / or any other information determined by the preprocessor 122 for generating prompt words for the generative AI model 132.

[0158] The prompter 124 generates prompt words to enable the generative AI model 132 to generate a QL query. For example... Figure 25 As shown, the prompter 124 receives a preprocessing signal 2518 and generates a prompt word 2520 based on the preprocessing signal 2518. In other words, the prompter comparator 2506, the pattern information determined by the pattern processor 2502, and / or any other information included in the preprocessing signal 2518. According to an embodiment, the prompter 124 includes a list of best practices for the query language in the prompt word 2520.

[0159] Generative AI model 132 generates QL queries from natural language based on prompts used to generate QL queries. For example, such as... Figure 25 As shown, the generative AI model 132 receives a prompt word 2520 and generates a QL query 2522 based on the prompt word. In this case, the QL query 2522 is an NL query 2510 adapted to be generated by... Figure 1 The database engine 134 performs a translation of the query language used when executing against database 114.

[0160] Post-processor 126 performs post-processing operations on the QL query generated by generative AI model 132. For example, post-processor 126 receives QL query 2522, performs post-processing operations on QL query 2522, and generates a response 2524 including the processed QL query. According to an embodiment, post-processor 126 provides response 2524 to application 120. In an alternative embodiment, the post-processor... Figure 1 Database engine 134 provides response 2524 to execute the processed QL query. Sample post-processing applications include, but are not limited to, parsing QL query 2522, repairing QL query 2522, generating a request for user feedback on QL query 2522, and displaying a request or window for providing user feedback in the UI of application 120 or another user device / application.

[0161] As described in this article, Figure 25 Examples of prompters 124 operate in various ways to generate prompt words to enable generative AI models 132 to generate QL queries based on NL queries (or other NL inputs). For example, Figure 26A flowchart 1500 of a process for generating a query is shown according to an example embodiment. According to the embodiment, the prompter 124 operates according to flowchart 2600. Note that not all steps of flowchart 2600 need to be performed in all embodiments. Based on the following... Figure 25 and Figure 26 Other structural and operational embodiments will be apparent to those skilled in the art as described herein.

[0162] Flowchart 2600 begins at step 2602. In step 2602, a second prompt word is generated based on the second natural language query and synthesized data. For example, such as regarding... Figure 25 As described, the prompter 124 generates the prompt word 2520 based on the NL query 2510 and the synthetic data selected by the embedding comparator 2506.

[0163] In step 2604, a second prompt word is provided to the LLM so that the LLM can convert the second natural language query into a second query language query. For example, regarding... Figure 25 As described, prompter 124 provides prompt words 2520 to generative AI model 132 so that generative AI model 132 can convert NL query 2510 (included in prompt words 2520) into QL query 2522.

[0164] As described in this article, the example of preprocessor 122 selects synthetic data, and Figure 25 The prompter 124 generates prompt words based on selected synthetic data in various ways. For example, Figure 27 A flowchart 2700 is shown illustrating a process for preprocessing prompts for a generative AI model according to an example embodiment. According to the embodiment, Figure 25 The language conversion engine 146 operates according to flowchart 2700. Note that not all steps of flowchart 2700 need to be performed in all embodiments. Figure 27 Compared to Figure 25 The following description, other structural and operational embodiments will be apparent to those skilled in the art.

[0165] Flowchart 2700 begins at step 2702. In step 2702, synthetic data is used to compare the query embedding describing the context of the second natural language query with the pair embedding describing the context of the first enhancement pair, the second enhancement pair, and / or the first correction pair. For example, as regarding Figure 25 As described, the embedding comparator 2506 compares the query embedding 2508 with the pair embedding, which describes the context of the pair (e.g., the enhancement pair or the correction pair) in the synthetic data 144 (in embodiments that include the synthetic data filter 2504, the filtered synthetic data 2516).

[0166] Step 2704 according to the embodiment is Figure 26 Another step in step 2604 of flowchart 2600. In step 2704, in response to the similarity between the query embedding and the pair embedding satisfying a similarity criterion, a second prompt word is generated based on the second natural language query and the first enhancement pair, the second enhancement pair, and / or the first correction pair. For example, as about Figure 25 As described, the prompter 124 generates a prompt word 2520 based on the NL query 2510 and the pairs included in the preprocessing signal 2518.

[0167] VI. Example Computer System Implementation The embodiments of synthetic data generation described herein are implemented in hardware or hardware in combination with one or more of software and / or firmware. Examples include: System 100, computing device 102, conversion server 104, embedded server 106, synthetic data server 108, model server 110, engine server 112, database 114, storage device 116, embedded model 128, synthetic data generator 130, generative AI model 132, database engine 134, language conversion engine 146, System 200, System 400, System 700, System 1300, System 1600, and System 18. The steps of system 200, system 2300, system 2500 and / or the components described therein, and / or flowcharts 300, 500, 600, 800, 900, 1000, 1100, 1200, 1400, 1700, 1900, 2000, 2100, 2200, 2400, 2600 and / or 2700 are each implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, the embedded model 128, the synthetic data generator 130, the generative AI model 132, the database engine 134, the language conversion engine 146, the system 200, the system 400, the system 700, the system 1300, the system 1600, the system 1800, the system 2300, the system 2500 and / or the components described therein, and / or the steps of flowcharts 300, 500, 600, 800, 900, 1000, 1100, 1200, 1400, 1700, 1900, 2000, 2100, 2200, 2400, 2600 and / or 2700 are implemented in one or more system-on-chips (SoCs). A SoC includes an integrated circuit chip that includes one or more of a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or additional circuitry, and optionally executes received program code and / or includes embedded firmware to perform functions.

[0168] The embodiments disclosed herein can be implemented in one or more mobile (mobile devices) and / or stationary (stationary devices) computing devices, and include any combination of features of such mobile and stationary computing devices. Examples of computing devices that can be implemented in the various embodiments are shown below. Figure 28 To describe. Figure 28 A block diagram of an exemplary computing environment 2800 including computing device 2802 is shown. Computing device 2802 is an example of computing device 102, conversion server 104, embedding server 106, synthetic data server 108, model server 110, and / or engine server 112, each of which includes one or more components of computing device 2802. In some embodiments, computing device 2802 is connected to devices outside computing environment 2800 via network 2804. Figure 28 (Not shown in the image) is communicatively coupled. Network 2804 is an example of network 118. Network 2804 includes one or more networks, such as a local area network (LAN), a wide area network (WAN), an enterprise network, the Internet, etc. In the example, network 2804 includes one or more wired and / or wireless components. In some examples, network 2804 additionally or alternatively includes a cellular network for cellular communications. Computing device 2802 is described in detail below.

[0169] Computing device 2802 can be any of a variety of types of computing devices. Examples of computing device 2802 include mobile computing devices such as handheld computers (e.g., personal digital assistants (PDAs)), laptop computers, tablet computers, hybrid devices, notebook computers, netbooks, mobile phones (e.g., cellular phones, smartphones, etc.), wearable computing devices (e.g., head-mounted augmented reality and / or virtual reality devices including smart glasses), or other types of mobile computing devices. In alternative examples, computing device 2802 is a fixed computing device such as a desktop computer, personal computer (PC), fixed server equipment, minicomputer, mainframe, supercomputer, etc. like Figure 28As shown, computing device 2802 includes various hardware and software components, including a processor 2810, storage device 2820, graphics processing unit (GPU) 2842, neural processing unit (NPU) 2844, multiple input devices 2830, multiple output devices 2850, multiple wireless modems 2860, one or more wired interfaces 2880, power supply 2882, location information (LI) receiver 2884, and accelerometer 2886. Memory 2820 includes memory 2856 and storage device 2888, with memory 2856 including non-removable memory 2822 and removable memory 2824. Memory 2820 also stores operating system 2812, applications 2814, and application data 2816. Wireless modem 2860 includes Wi-Fi modem 2862, Bluetooth modem 2864, and cellular modem 2866. Output device 2850 includes speaker 2852 and display 2854. Input device 2830 includes touchscreen 2832, microphone 2834, camera 2836, physical keyboard 2838, and trackball 2840. (Not...) Figure 28 All components of the computing device 2802 shown are present in all embodiments. Additional components, not shown, may be present, and in certain embodiments, any combination of components may exist. In the example, the components of the computing device 2802 are mounted to the circuit card (e.g., motherboard) of the computing device 2802, integrated into the housing of the computing device 2802, or otherwise included in the computing device 2802. The components of the computing device 2802 are described below.

[0170] In embodiments, a single processor 2810 (e.g., a central processing unit (CPU), microcontroller, microprocessor, signal processor, application-specific integrated circuit (ASIC), and / or other physical hardware processor circuitry) or multiple processors 2810 exist in computing device 2802 for performing tasks such as program execution, signal encoding, data processing, input / output processing, power control, and / or other functions. In examples, processor 2810 is a single-core or multi-core processor, and each processor core is single-threaded or multi-threaded (to provide multiple execution threads simultaneously). Processor 2810 is configured to execute program code stored in a computer-readable medium, such as program code of operating system 2812, and application program 2814 stored in memory 2820. The program code is constructed to cause processor 2810 to perform operations, including the processes / methods disclosed herein. Operating system 2812 controls the allocation and use of components of computing device 2802 and provides support for one or more application programs 2814 (also referred to as "applications" or "apps"). In the example, application 2814 includes common computing applications (e.g., email applications, calendars, contact managers, web browsers, messaging applications), other computing applications (e.g., word processing applications, map applications, media player applications, productivity suite applications), one or more machine learning (ML) models, and applications related to embodiments disclosed elsewhere herein. In the example, processor(s) 2810 includes one or more general-purpose processors (e.g., CPUs) configured with or coupled to one or more hardware accelerators (such as one or more NPUs 2844 and / or one or more GPUs 2842).

[0171] Any component in computing device 2802 may communicate with any other component according to its function, although not all connections are shown for ease of illustration. For example, such as Figure 28 As shown, bus 2806 is a multi-signal-line communication medium (e.g., conductive traces in silicon, metal traces along the motherboard, wires, etc.) present to communicatively couple processor 2810 to various other components of computing device 2802, although in other embodiments, alternative buses, additional buses, and / or one or more individual signal lines are present to communicatively couple components. Bus 2806 represents any one or more of several types of bus architectures, including memory buses or memory controllers, peripheral buses, accelerated graphics ports, and processor or local buses using any of the various bus architectures.

[0172] Storage device 2820 is a physical memory including one or both of memory 2856 and storage device 2888, which stores operating system 2812, application 2814, and application data 2816 according to any distribution. Non-removable memory 2822 includes one or more of random access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD), hard disk drive (e.g., a disk drive for reading from and writing to a hard disk), and / or other physical memory device types. In the example, non-removable memory 2822 includes main memory and is either separate from processor 2810 or fabricated in the same integrated circuit as processor 2810. Figure 28 As shown, non-removable memory 2822 stores firmware 2818 present to provide low-level control over the hardware. Examples of firmware 2818 include BIOS (Basic Input / Output System, such as in a personal computer) and boot firmware (e.g., in a smartphone). In the example, removable memory 2824 is inserted into a socket of computing device 2802 or otherwise coupled to computing device 2802 and can be removed by a user from computing device 2802. Removable memory 2824 can include any suitable type of removable memory device, including Secure Digital (SD) cards, Subscriber Identity Module (SIM) cards known in GSM communication systems, and / or other types of removable physical memory devices. In the example, one or more storage devices 2888 are present inside and / or outside the housing of computing device 2802 and may or may not be removable. Examples of storage devices 2888 include hard disk drives, SSDs, thumb drives (e.g., Universal Serial Bus (USB) flash drives), or other physical storage devices.

[0173] One or more programs are stored in memory 2820. Such programs include an operating system 2812, one or more application programs 2814, and other program modules and program data. Examples of such application programs include those for implementing a database 114, application 120, preprocessor 122, prompter 124, postprocessor 126, embedding model 128, synthetic data generator 130, generative AI model 132, database engine 134, language conversion engine 146, prompt word generator 202, synthetic data postprocessor 204, pair queue 402, embedding model interface 404, directory predictor 406, query parser 702, similarity evaluator 704, coverage evaluator 706, pair corrector 1602, synthetic data postprocessor 1604, feedback preprocessor 1802, feedback indicator 1804, syntax validator 1806, telemetry evaluator 1808, and self-corrector. Positive agent 1810, eligibility verifier 1812, prompter 1814, grammar evaluator 1816, consistency verifier 2302, consistency prompter 2304, consistency evaluator 2306, pattern processor 2502, synthetic data filter 2504, embedded comparator 2506 and / or the computer program logic (e.g., computer program code / instructions) of each component described herein, and any one of flowcharts 300, 500, 600, 800, 900, 1000, 1100, 1200, 1400, 1700, 1900, 2000, 2100, 2200, 2400, 2600 and / or 2700, and / or any individual step thereof.

[0174] The memory 2820 also stores data used and / or generated by the operating system 2812 and application 2814 as application data 2816. Examples of application data 2816 include web pages, text, images, tables, sound files, video data, and other data. In the example, application data 2816 is sent to and / or received from one or more network servers or other devices via one or more wired or wireless networks. The memory 2820 can be used to store additional data, including subscriber identifiers such as the International Mobile Subscriber Identity (IMSI) and device identifiers such as the International Mobile Equipment Identity (IMEI). Such identifiers can be sent to the network server to identify users and devices.

[0175] In the example, a user inputs commands and information into computing device 2802 through one or more input devices 2830 and receives information from computing device 2802 through one or more output devices 2850. The input devices 2830 include one or more of a touchscreen 2832, a microphone 2834, a camera 2836, a physical keyboard 2838, and / or a trackball 2840, and the output devices 2850 include one or more of a speaker 2852 and a display 2854. Each of the input devices 2830 and output devices 2850 is integrated into computing device 2802 (e.g., built into the housing of computing device 2802) or located externally to computing device 2802 (e.g., communicatively coupled to computing device 2802 via wired interfaces 2880 and / or wireless modems 2860 in a wired or wireless manner). Other input devices 2830 (not shown) may include natural user interfaces (NUIs), pointing devices (computer mice), joysticks, video game controllers, scanners, touchpads, styluses, voice recognition systems for receiving voice input, gesture recognition systems for receiving gesture input, etc. Other possible output devices (not shown) may include piezoelectric output devices or other haptic output devices. Some devices may be used for more than one input / output function. For example, a display 2854 may display information and may be operated as a touchscreen 2832 to serve as a user interface by receiving user commands and / or other information (e.g., by touch, finger gestures, virtual keyboard, etc.). Any number of each type of input device(s) 2830 and output device(s) 2850 may be present, including multiple microphones 2834, multiple cameras 2836, multiple speakers 2852, and / or multiple displays 2854.

[0176] In embodiments where GPU 2842 is present, GPU 2842 includes hardware (e.g., one or more integrated circuit chips implementing one or more of processing cores, multiprocessors, computing units, etc.) configured to accelerate computer graphics (two-dimensional (2D) and / or three-dimensional (3D)), perform image processing, and / or perform other parallel processing applications (e.g., training of neural networks, etc.). Examples of GPU 2842 performing computations related to 3D computer graphics include 2D acceleration and frame buffer capabilities, accelerating memory-intensive work such as texture mapping and rendering polygons, accelerating geometric computations (such as rotation and translation of vertices to different coordinate systems), supporting programmable shaders that manipulate vertices and textures, performing oversampling and interpolation techniques to reduce aliasing, and / or supporting very high-precision color spaces.

[0177] In the example, the NPU 2844 (also known as an "Artificial Intelligence (AI) Accelerator" or "Deep Learning Processor (DLP)") is a processor or processing unit configured to accelerate the execution of artificial intelligence and machine learning applications, such as the machine learning (ML) model (MLM) 2828. In this example, the NPU 2844 is configured for data-driven parallel computing and is efficient in processing large-scale multimedia data such as videos and images, as well as data for neural networks. The NPU 2844 is configured to efficiently handle AI-related tasks such as speech recognition, background blurring in video calls, and photo or video editing processes (such as object detection). In the embodiments of the ML model implementation disclosed herein, the NPU 2844 can be used to execute such an ML model, with the MLM 2828 being an example. According to the embodiments, the MLM 2828 is another example of the embedding model 128 and / or the generative AI model 132. For example, where applicable, the MLM 2828 is a generative AI model that generates complex, coherent, and / or original content. For example, a generative AI model can create complex sentences, lists, ranges, data tables, images, papers, etc. An example of a generative AI model is a language model. A language model is a model that estimates the probability of a lexical or sequence of lexical terms appearing in a longer sequence of lexical terms. In this context, a "lexical" is an atomic unit that the model is training and making predictions on. Examples of lexical terms include, but are not limited to, words, characters (e.g., alphanumeric characters, spaces, symbols, etc.), and subwords (e.g., root words, prefixes, or suffixes). In other types of models (e.g., image-based models), lexical terms can represent another type of atomic unit (e.g., a subset of images). Examples of language models applicable to the embodiments of this document include Large Language Models (LLMs), text-to-image AI image generation systems, text-to-video AI generation systems, and so on. A Large Language Model (LLM) is a language model with a large number of model parameters. In the examples, LLMs have millions, billions, trillions, or even more model parameters. The model parameters of an LLM are the weights and biases that the model learns during training. Some implementations of LLMs are transformer-based LLMs (e.g., the Generative Pre-trained Transformer (GPT) model family). A transformer is a neural network architecture that relies on a self-attention mechanism to transform an input embedding sequence into an output embedding sequence (e.g., without relying on convolutional or recurrent neural networks).

[0178] In another example, the NPU 2844 is used to train the MLM 2828. To train the MLM 2828, training data is collected, including input features (attributes) and their corresponding output labels / target values ​​(e.g., for supervised learning). The training algorithm is the computational process used to enable the MLM 2828 to learn from the training data. Parameters / weights are internal settings of the MLM 2828, which are adjusted by the training algorithm during training to reduce the discrepancy between the MLM 2828's predictions and actual results (e.g., output labels). In some examples, the MLM 2828 is set with initial values ​​for its parameters / weights. A loss function measures the difference between the MLM 2828's predictions and target values, and the MLM 2828's parameters / weights are adjusted to minimize the loss function. The parameters / weights are iteratively adjusted using optimization techniques such as gradient descent. In this way, the MLM 2828 is generated through training the NPU 2844 to generate inference based on a received set of input features for a specific application. MLM 2828 is generated as a computer program or other type of algorithm configured to generate output (e.g., classification, prediction / inference) based on received input features and stored in the form of a file or other data structure.

[0179] In the example, this training of the MLM 2828 by the NPU 2844 can be supervised or unsupervised. According to supervised learning, the MLM 2828 is trained with input objects (e.g., a vector of predictor variables) and desired output values ​​(e.g., human-labeled supervisory signals). The training data is processed, and a function is constructed that maps new data to the expected output values. In a particular implementation, example algorithms that the NPU 2844 can use to perform supervised training of the MLM 2828 include support vector machines, linear regression, logistic regression, Naive Bayes, linear discriminant analysis, decision trees, K-nearest neighbors, neural networks, and similarity learning.

[0180] In the example of supervised learning of LLM 2828, MLM 2828 can be trained by exposing the LLM to (e.g., a large amount) of text (e.g., pre-defined datasets, books, articles, text-based dialogues, web pages, transcripts, forum entries, and / or any other form of text and / or combinations thereof). In the example, training data is provided from a database, from the Internet, from a system, etc. Furthermore, the LLM can be fine-tuned using reinforcement learning with human feedback (RLHF), where the LLM is provided with the same input twice and two different outputs are provided, with the user ranking which output is preferred. In this case, the user's ranking is used to improve the model. Additionally, in the example embodiment, the LLM is trained to perform in various ways, such as as a completion model (a model that provides several words or lexical units and generates words or lexical units to follow the input), as a dialogue model (a model that provides answers to dialogue prompts or other types of responses), as a combination of completion and dialogue models, or as another type of LLM model.

[0181] According to unsupervised learning, the MLM 2828 is trained to learn patterns from unlabeled data. For example, in an embodiment where the MLM 2828 implements unsupervised learning techniques, the MLM 2828 identifies one or more categories or clusters to which the input belongs. During the training phase of the MLM 2828 based on unsupervised learning, the MLM 2828 attempts to mimic the provided training data and uses the errors in its mimicked output to correct itself (i.e., correct the weights and biases). In another example, the NPU 2844 performs unsupervised training of the MLM 2828 according to one or more alternative techniques, such as the Hopfield learning rule, the Boltzmann learning rule, contrastive divergence, wake-sleep, variational inference, maximum likelihood, maximum a posteriori, Gibbs sampling, and backpropagation to reconstruct errors or reparameterize hidden states.

[0182] Note that the NPU 2844 is not necessarily present in all ML model implementations. In implementations where an ML model is present, any one or more of the processor 2810, GPU 2842, and / or NPU 2844 may be present to train and / or execute the MLM2828.

[0183] One or more wireless modems 2860 may be coupled to one or more antennas (not shown) of computing device 2802 and may enable bidirectional communication between processor 2810 and devices external to computing device 2802 via network 2804, as will be understood by those skilled in the art. Wireless modems 2860 are generally shown and may include cellular modems 2866 for communicating with one or more cellular networks, such as GSM networks for data and voice communication within a single cellular network, between cellular networks, or between a mobile device and the Public Switched Telephone Network (PSTN). In the example, wireless modems 2860 may also include or alternatively include other radio-based modem types, such as Bluetooth modem 2864 (also referred to as a “Bluetooth device”) and / or Wi-Fi modem 2862 (also referred to as a “wireless adapter”). Wi-Fi modem 2862 is configured to communicate with access points or other remotely Wi-Fi-enabled devices according to one or more wireless network protocols based on the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards (typically used for local area network connections and Internet access of devices). The Bluetooth modem 2864 is configured to communicate with another Bluetooth-enabled device in accordance with Bluetooth Short Range Wireless Technology standards such as IEEE 802.15.1 and / or multiple Bluetooth standards managed by the Bluetooth Special Interest Group (SIG).

[0184] The computing device 2802 may also include a power supply 2882, an LI receiver 2884, an accelerometer 2886, and / or (multiple) wired interfaces 2880. Example wired interfaces 2880 include a USB port, an IEEE 1394 (FireWire) port, an RS-232 port, a High-Definition Multimedia Interface (HDMI) port (e.g., for connection to an external display), a DisplayPort port (e.g., for connection to an external display), an audio port, and / or an Ethernet port, the purpose and function of each of which are well known to those skilled in the art. The wired interface 2880 of the computing device 2802 provides a wired connection between the computing device 2802 and a network 2804, or provides a wired connection between the computing device 2802 and one or more devices / peripherals when such a device / peripheral is external to the computing device 2802 (e.g., a pointing device, a display 2854, a speaker 2852, a camera 2836, a physical keyboard 2838, etc.). Power supply 2882 is configured to supply power to each component of computing device 2802 and receive power from a battery inside computing device 2802 and / or from a power cord inserted into a power port (e.g., USB port, A / C power port) of computing device 2802. LI receiver 2884 can be used for location determination of computing device 2802 and, in the example, includes a satellite navigation receiver such as a Global Positioning System (GPS) receiver and / or other types of location determiners configured to determine the location of computing device 2802 based on received information (e.g., using cell tower triangulation, etc.). Accelerometer 2886 (if present) is configured to determine the orientation of computing device 2802.

[0185] Note that the components shown for computing device 2802 are not essential or all-encompassing, and fewer or more components may be present, as those skilled in the art will recognize. In the example, computing device 2802 includes one or more of a gyroscope, barometer, proximity sensor, ambient light sensor, digital compass, etc. In the example, processor 2810 and memory 2856 coexist in the same semiconductor device package, such as being included together in an integrated circuit chip, FPGA, or system-on-a-chip (SOC), optionally along with other components of computing device 2802.

[0186] In this embodiment, computing device 2802 is configured to implement any of the features described above in the flowchart herein. Computer program logic for performing any operations, steps, and / or functions described herein is stored in storage device 2820 and executed by processor 2810.

[0187] In some embodiments, server infrastructure 2870 resides within computing environment 2800 and is communicatively coupled to computing device 2802 via network 2804. Server infrastructure 2870 (when present) is a set of network-accessible servers (e.g., a cloud-based environment or platform). Figure 28 As shown, server infrastructure 2870 includes clusters 2872. Each cluster 2872 includes a group of one or more compute nodes and / or a group of one or more storage nodes. For example, as Figure 28 As shown, cluster 2872 includes nodes 2874. Each node 2874 can be accessed via network 2804 (e.g., in a "cloud-based" embodiment) to build, deploy, and manage applications and services. In the example, any node 2874 is a storage node comprising multiple physical storage disks, SSDs, and / or other physical storage devices accessible via network 2804 and configured to store data associated with applications and services managed by node 2874.

[0188] Each node 2874, serving as a computing node, includes one or more server computers, server systems, and / or computing devices. For example, node 2874 according to an embodiment includes one or more components of the computing device 2802 disclosed herein. Each node 2874 is configured to execute one or more software applications (or "applications") and / or service and / or manage hardware resources (e.g., processors, memory, etc.) used by users (e.g., clients) of a set of network-accessible servers. In the example, as... Figure 28 As shown, node 2874 includes node 2846, which includes storage device 2848 and / or (multiple) processors 2858 (e.g., processor 2810, GPU 2842, and / or NPU 2844, similar to computing device 2802). Storage device 2848 stores application 2876 and application data 2878. (Multiple) processors 2858 operate applications 2876 that access and / or generate related application data 2878. In implementations, node operation of node 2846, such as node 2874, may include one or more virtual machines, each virtual machine emulating a system architecture (e.g., an operating system) in an isolated manner, on which applications such as application 2876 are executed.

[0189] In embodiments, one or more of clusters 2872 are located / co-located (e.g., housed together in one or more nearby buildings with associated components such as backup power, redundant data communications, environmental controls, etc.) to form a data center, or are otherwise arranged. Therefore, in embodiments, one or more of clusters 2872 are included in a distributed collection of data centers within a data center. In embodiments, exemplary computing environment 2800 includes part of a cloud-based platform.

[0190] In an embodiment, computing device 2802 accesses application 2876 to execute in any manner, such as by a client application and / or browser at computing device 2802.

[0191] In the example, for network (e.g., cloud) backup and data security purposes, computing device 2802 additionally and / or alternatively synchronizes copies of application 2814 and / or application data 2816 to be stored as application 2876 and / or application data 2878 at network-based server infrastructure 2870. In the example, operating system 2812 and / or application 2814 include a file hosting service client configured to synchronize applications and / or data stored in storage 2820 at network-based server infrastructure 2870.

[0192] In some embodiments, a local server 2892 resides within a computing environment 2800 and is communicatively coupled to computing device 2802 via network 2804. The local server 2892 (when present) is hosted within the organization's infrastructure and, in many cases, physically hosted on-site at the organization's facilities. The local server 2892 is controlled, managed, and maintained by the organization's information technology (IT) personnel or its IT partners. Application data 2898 can be shared by the local server 2892 among the organization's computing devices (including computing device 2802 (when part of the organization)) via the organization's local network and / or other networks accessible to the organization (including the Internet). Furthermore, in this example, the local server 2892 provides applications such as application 2896 to the organization's computing devices (including computing device 2802). Therefore, in this example, the local server 2892 includes a storage device 2894 (which includes one or more physical storage devices, such as storage disks and / or SSDs) for storing application 2896 and application data 2898, and includes a processor 2890 (e.g., a processor 2810, GPU 2842, and / or NPU 2844 similar to computing device 2802) for executing application 2896. In some embodiments, multiple processors 2890 are present for executing application 2896 and / or for other purposes. In a further example, computing device 2802 is configured to synchronize copies of application 2814 and / or application data 2816 as backup storage of application 2896 and / or application data 2898 at the local server 2892.

[0193] The embodiments described herein can be implemented in one or more of computing device 2802, network-based server infrastructure 2870, and local server 2892. For example, in some embodiments, computing device 2802 is used to implement a system, client, or device or a component / subcomponent thereof disclosed elsewhere herein. In other embodiments, a combination of computing device 2802, network-based server infrastructure 2870, and / or local server 2892 is used to implement a system, client, or device or a component / subcomponent thereof disclosed elsewhere herein.

[0194] As used herein, the terms “computer program medium,” “computer-readable medium,” “computer-readable storage medium,” and “computer-readable storage device,” etc., refer to physical hardware media. Examples of such physical hardware media include any hard disk, optical disk, SSD, other physical hardware media such as RAM, ROM, flash memory, digital video disk, zip disk, microelectromechanical (MEMS) memory, nanotechnology-based storage devices, and other types of physical / tangible hardware storage media of Memory 2820. Such computer-readable media and / or storage media are distinct from and do not overlap with communication media, propagating signals, and signals themselves. In other words, “computer program medium,” “computer-readable medium,” “computer-readable storage medium,” and “computer-readable storage device” do not include communication media, propagating signals, and signals themselves. Communication media embody computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves. The term “modulated data signal” refers to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, communication media include wireless media such as acoustic, RF, infrared, and other wireless media, as well as wired media. The embodiments also relate to such a communication medium that is separate from and does not overlap with the embodiments relating to computer-readable storage media.

[0195] As described above, computer programs and modules (including application program 2814) are stored in memory 2820. Such computer programs can also be received via network 2804 through wired interfaces 2860 and / or wireless modems 2860. When executed or loaded by an application, such computer programs enable computing device 2802 to implement the features of the embodiments discussed herein. Therefore, such computer programs represent the controller of computing device 2802.

[0196] The embodiments also relate to computer program products comprising computer code or instructions stored on any computer-readable medium or computer-readable storage medium. Such computer program products include physical storage of memory 2820 and other types of physical storage.

[0197] VII. Additional Exemplary Examples This paper describes a system for generating synthetic data. The system includes processor circuitry and a memory device. The memory device stores program code to be executed by the processor circuitry. This program code includes a synthetic data generator.

[0198] In another aspect of the aforementioned system, the synthetic data generator: obtains a dataset pair including a first natural language query and a first query language query; uses the dataset pair and first prediction directory information to generate a first prompt word, so that the generative AI model generates a variant of the dataset pair; in response to providing the first prompt word to the generative AI model, receives a first augmentation pair including a first augmented natural language query and a first augmented query language query, wherein the first augmented natural language query is a variant of the first natural language query and the first augmented query language query is a variant of the first query language query; and generates synthetic data including the first augmentation pair.

[0199] In another aspect of the aforementioned system, a synthetic data generator receives database embeddings and pairs of embeddings, the database embeddings describing the context of a portion of the database, and the pairs of embeddings describing the context of a dataset pair; and determines first predicted catalog information based on the similarity between the pairs of embeddings and subsets of the database embeddings.

[0200] In another aspect of the aforementioned system, a synthetic data generator provides dataset pairs to an embedding model, which is configured to generate embeddings based on the input data; and receives embedding pairs from the embedding model.

[0201] In another aspect of the aforementioned system, the synthetic data generator: provides a first cue word to the generative AI model to generate a set of enhancement pairs including a first enhancement pair and a second enhancement pair, the second enhancement pair including a second enhanced natural language query and a second enhanced query language query, the second enhanced natural language query being another variant of the first natural language query and the second enhanced query language query being another variant of the first query language query; filters the second enhancement pair from the set of enhancement pairs to generate a filtered set of pairs; and generates synthetic data including the filtered set of pairs.

[0202] In another aspect of the aforementioned system, the synthetic data generator filters the second enhancement pair by providing a second enhanced query language query to a query parser configured to determine whether the query is valid; and by receiving an indication that the second enhanced query language query is invalid.

[0203] In another aspect of the aforementioned system, the synthetic data generator filters the second enhancement pair by determining that the similarity between the second enhancement pair and at least one of the following satisfies a similarity criterion: the first dataset pair, the first enhancement pair, or another pre-existing dataset pair.

[0204] In another aspect of the aforementioned system, the second enhancement pair corresponds to a portion of data in the database, and in order to filter the second enhancement pair, the synthetic data generator determines that the portion is covered by the dataset pair corresponding to that portion, which satisfies the coverage criterion.

[0205] In another aspect of the aforementioned system, the synthetic data generator: determines that the coverage of a portion of the data in the database does not meet the coverage criteria; and uses a generative AI model to generate a second enhancement pair based on at least one of the first dataset pair or the first enhancement pair.

[0206] In another aspect of the aforementioned system, the synthetic data generator: uses the first enhancement pair and the second prediction directory information to generate a second prompt word, so that the generative AI model generates a variant of the first enhancement pair; in response to providing the second prompt word to the generative AI model, it receives a second enhancement pair including a second enhanced natural language query and a second enhanced query language query, the second enhanced natural language query being a variant of the first enhanced natural language query and the second enhanced query language query being a variant of the first enhanced query language query; and generates synthetic data including the first enhancement pair and the second enhancement pair.

[0207] In another aspect of the aforementioned system, the synthetic data generator: receives a negative feedback instruction for a first query language query generated by a generative AI model based on a first natural language query; generates a correction pair based on the instruction and the first natural language query, the correction pair including the first natural language query and a corrected query language query, the corrected query language query being a syntactically valid transformation of the first natural language query; determines that the correction pair meets the criteria for synthetic data storage; and stores the verified correction pair as synthetic data in the synthetic data storage.

[0208] In another aspect of the aforementioned system, in order to prompt the generative AI model to generate correction pairs, the synthetic data generator: generates a first prompt word based on the instruction and a first natural language query to enable the generative AI model to generate candidate pairs; determines that the candidate pairs are grammatically invalid; generates a second prompt word to enable the generative AI model to generate correction pairs based on the instruction, the first natural language query, and the determination that the candidate pairs are grammatically invalid; and determines that the correction pairs are grammatically valid.

[0209] In another aspect of the aforementioned system, in order to generate a correction pair, the synthetic data generator: identifies a second query language query executed against the database; determines that the second query language query corresponds to a first natural language query; and generates a correction pair based on the first natural language query and the second query language query.

[0210] In another aspect of the aforementioned system, in order to determine whether the calibration pairs meet the criteria for synthetic data storage, the synthetic data generator: generates prompt words to enable the generative AI model to generate candidate natural language queries based on the calibration query language query; and determines whether the similarity between the first natural language query and the candidate natural language queries meets the consistency criteria.

[0211] In another aspect of the aforementioned system, in order to determine whether the calibration pair meets the criteria for synthetic data storage, the synthetic data generator determines that the similarity between the calibration pair and the dataset pair stored in the synthetic data storage does not meet the similarity criteria, the dataset pair including synthetic natural language queries and synthetic query language queries.

[0212] In another aspect of the aforementioned system, the calibration pair corresponds to a portion of data in the database, and in order to determine whether the calibration pair meets the criteria for synthetic data storage, the synthetic data generator determines that the coverage of the dataset pair corresponding to that portion does not meet the coverage criteria.

[0213] In another aspect of the aforementioned system, in order to generate a correction pair, the synthetic data generator determines the eligibility of the first natural language query to be converted based on at least one of the user account's permissions or an available table in the database.

[0214] In another aspect of the aforementioned system, the synthetic data generator enables the language conversion engine to use synthetic data to generate second prompt words based on the second natural language query and the synthetic data, and to provide the second prompt words to the generative AI model so that the generative AI model can convert the second natural language query into a second query language query.

[0215] In another aspect of the aforementioned system, in order for the language conversion engine to generate a second prompt word using synthetic data, the synthetic data generator causes the language conversion engine to use synthetic data to compare the query embedding of the context describing the second natural language query with the pair embedding of the context describing the first enhancement pair, and to generate a second prompt word based on the second natural language query and the first enhancement pair in response to the similarity between the query embedding and the pair embedding satisfying a similarity criterion.

[0216] On the other hand, generative AI models are large language models.

[0217] This paper describes a first aspect of a method for generating synthetic data. In this respect, the method includes: obtaining a dataset pair comprising a first natural language query and a first query language query; using the dataset pair and first prediction directory information to generate a first prompt word, such that a generative AI model generates a variant of the dataset pair; in response to providing the first prompt word to the generative AI model, receiving a first augmentation pair comprising a first augmented natural language query and a first augmented query language query, wherein the first augmented natural language query is a variant of the first natural language query and the first augmented query language query is a variant of the first query language query; and generating synthetic data comprising the first augmentation pair.

[0218] In another implementation of the first aspect of the method for generating synthetic data, the method further includes: receiving a database embedding and a pair of embeddings, the database embedding describing the context of a portion of a database and the pair of embeddings describing the context of a dataset pair; and determining first predictive catalog information based on the similarity between the pair of embeddings and a subset of the database embeddings.

[0219] In another implementation of the first aspect of the method for generating synthetic data, the method further includes: providing dataset pairs to an embedding model configured to generate embeddings based on input data; and receiving embedding pairs from the embedding model.

[0220] In another implementation of the first aspect of the method for generating synthetic data, the method further includes: providing a first cue word to a generative AI model to generate a set of enhancement pairs including a first enhancement pair and a second enhancement pair, the second enhancement pair including a second enhanced natural language query and a second enhanced query language query, the second enhanced natural language query being another variant of the first natural language query and the second enhanced query language query being another variant of the first query language query; filtering the second enhancement pair from the set of enhancement pairs to generate a filtered set of pairs; and generating synthetic data including the filtered set of pairs.

[0221] In another implementation of the first aspect of the method for generating synthetic data, filtering the second enhancement pair includes: providing a second enhanced query language query to a query parser configured to determine whether the query is valid; and receiving an indication that the second enhanced query language query is invalid.

[0222] In a further implementation of a first aspect of the method for generating synthetic data, filtering the second enhancement pair includes: determining that the similarity between the second enhancement pair and at least one of the following satisfies a similarity criterion: a first dataset pair, the first enhancement pair, or another pre-existing dataset pair.

[0223] In another implementation of the first aspect of the method for generating synthetic data, the second enhancement pair corresponds to a portion of data in the database, and filtering the second enhancement pair includes: determining that the portion is covered by a coverage criterion of the dataset pair corresponding to the portion.

[0224] In another implementation of the first aspect of the method for generating synthetic data, the method further includes: determining that a portion of the data in the database does not satisfy a coverage criterion; and generating a second enhancement pair based on at least one of a first dataset pair or a first enhancement pair using a generative AI model.

[0225] In another implementation of the first aspect of the method for generating synthetic data, the method further includes: generating a second cue word using a first enhancement pair and second prediction directory information to enable a generative AI model to generate a variant of the first enhancement pair; receiving a second enhancement pair including a second enhanced natural language query and a second enhanced query language query, the second enhanced natural language query being a variant of the first enhanced natural language query and the second enhanced query language query being a variant of the first enhanced query language query, in response to providing the second cue word to the generative AI model; and generating synthetic data including the first enhancement pair and the second enhancement pair.

[0226] This paper describes a second aspect of a method for generating synthetic data. In this respect, the method includes: receiving an instruction for negative feedback on a first query language query generated by a generative AI model based on a first natural language query; generating a correction pair based on the instruction and the first natural language query, the correction pair including the first natural language query and a corrected query language query, the corrected query language query being a syntactically valid transformation of the first natural language query; determining that the correction pair satisfies criteria for synthetic data storage; and storing the verified correction pair as synthetic data in the synthetic data storage.

[0227] In another implementation of the second aspect of the method for generating synthetic data, generating correction pairs includes: prompting a generative AI model to generate correction pairs based on an instruction and a first natural language query.

[0228] In another implementation of a second aspect of the method for generating synthetic data, the step of prompting the generative AI model to generate correction pairs includes: generating a first prompt word to enable the generative AI model to generate candidate pairs based on the instruction and a first natural language query; determining that the candidate pairs are grammatically invalid; generating a second prompt word to enable the generative AI model to generate correction pairs based on the instruction, the first natural language query, and the determination that the candidate pairs are grammatically invalid; and determining that the correction pairs are grammatically valid.

[0229] In another implementation of a second aspect of the method for generating synthetic data, generating a correction pair includes: identifying a second query language query performed against a database; determining that the second query language query corresponds to a first natural language query; and generating a correction pair based on the first natural language query and the second query language query.

[0230] In another implementation of the second aspect of the method for generating synthetic data, determining that the calibration pairs satisfy the criteria for storing synthetic data includes: generating prompt words to enable a generative AI model to generate candidate natural language queries based on the calibration query language query; and determining that the similarity between the first natural language query and the candidate natural language queries satisfies the consistency criterion.

[0231] In another implementation of the second aspect of the method for generating synthetic data, determining that the calibration pair satisfies the criteria for the synthetic data storage includes: determining that the similarity between the calibration pair and a dataset pair stored in the synthetic data storage does not satisfy the similarity criteria, the dataset pair including a synthetic natural language query and a synthetic query language query.

[0232] In another implementation of the second aspect of the method for generating synthetic data, the calibration pair corresponds to a portion of data in a database, and determining that the calibration pair satisfies the criteria for storing synthetic data includes: determining that the coverage of the dataset pair corresponding to that portion does not satisfy the coverage criteria.

[0233] In another embodiment of the second aspect of the method for generating synthetic data, generating the correction pair includes determining the eligibility of a first natural language query to be converted based on user account permissions or at least one item from an available table in a database.

[0234] In a first or second aspect of the method for generating synthetic data, the method further includes: enabling a language conversion engine to utilize the synthetic data to: generate a second cue word based on a second natural language query and the synthetic data, and providing the second cue word to a generative AI model so that the generative AI model converts the second natural language query into a second query language query.

[0235] In a first or second aspect of the method for generating synthetic data, enabling a language conversion engine to generate a second prompt word using the synthetic data includes: enabling the language conversion engine to: use the synthetic data to compare a query embedding describing the context of a second natural language query with a pair embedding describing the context of a first augmentation pair, and generate a second prompt word based on the second natural language query and the first augmentation pair in response to a similarity criterion being satisfied between the query embedding and the pair embedding.

[0236] In the first or second aspect of the methods used to generate synthetic data, generative AI models are large language models.

[0237] A computer-readable storage medium encoded with program instructions that, when executed by processor circuitry, perform any of the aforementioned methods described herein.

[0238] In another embodiment of any of the foregoing systems, methods and / or computer-readable storage media, generative AI models other than LLM are utilized.

[0239] VIII. Conclusion References to "an embodiment," "embodiment," "example embodiment," etc., in the specification indicate that the described embodiment may include a particular feature, structure, or characteristic, but each embodiment may not necessarily include that particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Additionally, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is believed that in conjunction with other embodiments, whether explicitly described or not, the influence of such feature, structure, or characteristic on it is within the knowledge of those skilled in the art.

[0240] In this discussion, unless otherwise stated, adjectives modifying one or more features of an implementation of this disclosure, describing a condition or relational characteristic, should be understood to indicate that the condition or characteristic is limited to acceptable tolerances for the operation of the implementation for its intended purpose. Furthermore, if the execution of an operation is described herein as "in response to" one or more factors, it should be understood that the one or more factors can be considered as the sole contributing factor to the occurrence of the operation, or as contributing factors together with one or more additional factors, and that the operation can occur at or after the establishment of the one or more factors. Moreover, in the use of "based on" to indicate that an effect is a result of the indicated cause, it should be understood that the effect need not be produced solely by the indicated cause, and any number of possible additional causes may contribute to the production of the effect. Therefore, as used herein, the term "based on" should be understood to be equivalent to the term "at least based on".

[0241] Numerous example embodiments have been described above. Any section / headings provided herein are not intended to be limiting. Embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments disclosed in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0242] Furthermore, example embodiments have been described above with respect to one or more running examples. Such running examples describe one or more specific implementations of the example embodiments; however, the embodiments described herein are not limited to these specific implementations.

[0243] Furthermore, example embodiments of LLM have already been described; however, this document also envisions embodiments that can utilize other types of generative AI models (e.g., Generative Adversarial Networks (GANs)). Moreover, implementations of the described systems and / or methods can utilize multimodal models with more than one modality of input and / or output. For example, alternative embodiments utilize multimodal generative AI models that generate text and images based on prompts.

[0244] Furthermore, according to the described embodiments and techniques, any component of a system, computing device, server, application, embedded model, synthetic data generator, generative AI model, database engine, language conversion engine, memory, and / or its functionality can be activated for its operation / execution based on other operations, functions, actions, etc. (including initialization, completion, and / or execution of operations, functions, actions, etc.).

[0245] In some example embodiments, one or more operations of the flowchart described herein may not be performed. Furthermore, operations other than or replacing the operations of the flowchart described herein may be performed. Additionally, in some example embodiments, one or more operations of the flowchart described herein may be performed out of order, in an alternative order, or partially (or completely) simultaneously with each other or with other operations.

[0246] The embodiments described herein and / or any other systems, subsystems, devices and / or components disclosed herein may be implemented in hardware (e.g., hardware logic / circuit) or any combination of hardware and software (computer program code configured to execute in one or more processors or processing devices) and / or firmware.

[0247] While various embodiments have been described above, it should be understood that they are presented by way of example only and not as limitation. It will be apparent to those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the embodiments. Therefore, the breadth and scope of the embodiments should not be limited by any of the exemplary embodiments described above, but should be defined solely by the appended claims and their equivalents.

Claims

1. A system (100, 200, 400, 700, 2800) for generating synthetic data (144, 214) for use in performance benchmarking, comprising: Processor circuits (2810, 2858, 2890); as well as Memory devices (2820, 2848, 2856, 2888, 2894) storing program code (2814, 2876, 2896) to be executed by the processor circuitry (2810, 2858, 2890), the program code (2814, 2876, 2896) comprising: Synthetic data generator (130), which: Obtain the dataset pair (206, 412) that includes the first natural language query and the first query language query; The dataset pair (206, 412) and the first prediction directory information (208) are used to generate a first prompt word (210) so that the large language model (LLM) (132) generates a variant of the dataset pair; In response to providing the first prompt word (210) to the LLM (132), a first enhancement pair (212, 710) is received, comprising a first enhanced natural language query and a first enhanced query language query, wherein the first enhanced natural language query is a variant of the first natural language query, and the first enhanced query language query is a variant of the first query language query; and Generate synthetic data (144, 214) including the first enhancement pair (212).

2. The system according to claim 1, wherein the synthetic data generator further comprises: The language conversion engine uses the synthesized data to: A second prompt word is generated based on the second natural language query and the synthesized data; and The second prompt word is provided to the LLM so that the LLM can convert the second natural language query into a second query language query.

3. The system of claim 2, wherein, in order for the language conversion engine to use the synthesized data to generate the second prompt word, the synthesized data generator further: Enable the language conversion engine: The synthesized data is used to compare the query embedding describing the context of the second natural language query with the pair embedding describing the context of the first augmented pair, and In response to the similarity between the query embedding and the pair embedding satisfying the similarity criterion, the second prompt word is generated based on the second natural language query and the first enhanced pair.

4. The system of claim 2, wherein the second prompt word is generated based on the first enhancement pair, and the program code further includes a correction component, the correction component: Receive an indication of negative feedback for the query in the second query language; A correction pair is generated based on the instruction and the second natural language query. The correction pair includes the second natural language query and a corrected query language query, wherein the corrected query language query is a syntactically valid transformation of the second natural language query. Update the synthetic data to include the correction pair.

5. The system of claim 4, wherein, in order to generate the calibration pair, the calibration component: The LLM is prompted to generate the correction pair based on the instruction and the second natural language query.

6. The system of claim 5, wherein, in order to prompt the LLM to generate the calibration pair, the calibration pair component: Generate a second prompt word so that the LLM generates candidate pairs based on the prompt and the second natural language query; The candidate pair was determined to be syntactically invalid. A second cue word is generated so that the LLM generates the correction pair based on the cue, the second natural language query, and the determination that the candidate pair is grammatically invalid. as well as Determine that the correction pair is grammatically valid.

7. The calibration system of claim 4, wherein, in order to generate the calibration pair, the calibration component: Identifies a third-language query performed against the database; Determine that the third query language query corresponds to the second natural language query; and The correction pair is generated based on the second natural language query and the third query language query.

8. The system of claim 1, wherein the synthetic data generator further comprises: Receive a database embedding and a pair of embeddings, wherein the database embedding describes the context of a portion of the database, and the pair of embeddings describes the context of the dataset pair; and The first predicted directory information is determined based on the similarity between the pair of embeddings and a subset of the database embeddings.

9. The system of claim 8, wherein the synthetic data generator further comprises: The dataset pairs are provided to an embedding model, which is configured to generate embeddings based on the input data; and Receive the pair of embeddings from the embedding model.

10. The system of claim 1, wherein the synthetic data generator further comprises: The first prompt word is provided to the LLM to enable the LLM to generate a set of enhancement pairs including the first enhancement pair and the second enhancement pair, wherein the second enhancement pair includes a second enhanced natural language query and a second enhanced query language query, wherein the second enhanced natural language query is another variant of the first natural language query and the second enhanced query language query is another variant of the first query language query; Filter the second enhancement pair from the set of enhancement pairs to generate a filtered set of pairs; and Generate the synthetic data that includes the filtered set of pairs.

11. The system of claim 1, wherein the synthetic data generator further comprises: It was determined that the coverage of a portion of the data in the database did not meet the coverage criteria; and The LLM is used to generate a second enhancement pair based on at least one of the first dataset pair or the first enhancement pair.

12. The system of claim 1, wherein the synthetic data generator further comprises: The first enhancement pair and the second predicted directory information are used to generate a second cue word so that the LLM generates a variant of the first enhancement pair; In response to providing the second prompt word to the LLM, a second enhancement pair is received, comprising a second enhanced natural language query and a second enhanced query language query, wherein the second enhanced natural language query is a variant of the first enhanced natural language query and the second enhanced query language query is a variant of the first enhanced query language query; as well as Generate the synthetic data including the first enhancement pair and the second enhancement pair.

13. A method (300) for generating synthetic data for use in performance benchmarking, comprising: Obtain a dataset pair including the first natural language query and the first query language query (302); The dataset pair and the first prediction directory information are used to generate a first prompt word so that the generative artificial intelligence (AI) model generates a variant of the dataset pair (304); In response to providing the first prompt word to the generative AI model, a first enhancement pair including a first enhanced natural language query and a first enhanced query language query, wherein the first enhanced natural language query is a variant of the first natural language query and the first enhanced query language query is a variant of the first query language query (306); and Generate synthetic data including the first enhancement pair (308).

14. The method of claim 13, further comprising: The language conversion engine uses the synthesized data to: A second prompt word is generated based on the second natural language query and the synthesized data; as well as The second prompt word is provided to the generative AI model so that the generative AI model can convert the second natural language query into a second query language query.

15. The method of claim 14, wherein the second prompt word is generated based on the first enhancement pair, and the method further comprises: Receive an indication of negative feedback for the query in the second query language; A correction pair is generated based on the instruction and the second natural language query. The correction pair includes the second natural language query and a corrected query language query, wherein the corrected query language query is a syntactically valid transformation of the second natural language query. Update the synthetic data to include the correction pair.

16. The method of claim 15, wherein, in order to generate the calibration pair, the calibration component: The generative AI model is prompted to generate the correction pair based on the instruction and the second natural language query.

17. The method of claim 13, further comprising: The first prompt word is provided to the generative AI model so that the generative AI model generates a set of enhancement pairs including the first enhancement pair and the second enhancement pair, wherein the second enhancement pair includes a second enhanced natural language query and a second enhanced query language query, wherein the second enhanced natural language query is another variant of the first natural language query and the second enhanced query language query is another variant of the first query language query. The second enhancement pair is filtered from the set of enhancement pairs to generate a filtered set of pairs in the following manner: The second enhanced query language query is provided to the query parser, which is configured to determine whether the query is valid and to receive an indication that the second enhanced query language query is invalid. The similarity between the second enhancement pair and at least one of the following satisfies the similarity criterion: the first dataset pair, the first enhancement pair, or another pre-existing dataset pair; or A portion of the data in the database is determined to be covered by a dataset pair corresponding to that portion, satisfying the coverage criterion; the portion corresponds to the second enhancement pair. as well as Generate the synthetic data that includes the filtered set of pairs.

18. A computer-readable storage medium (2820) encoded with program instructions (2814), said program instructions (2814) executing a method (300) when executed by a processor circuit (2810), said method (300) comprising: Obtain a dataset pair including the first natural language query and the first query language query (302); The dataset pair and the predicted directory information are used to generate a first prompt word so that the Large Language Model (LLM) generates a variant of the dataset pair (304); In response to providing the first prompt word to the LLM, an enhancement pair including an enhanced natural language query and an enhanced query language query is received, the enhanced natural language query being a variant of the first natural language query and the enhanced query language query being a variant of the first query language query (306); as well as Enable the language conversion engine to utilize the aforementioned enhancements to: The second prompt word (2602) is generated based on the second natural language query and the aforementioned enhancement. as well as The second prompt word is provided to the LLM so that the LLM converts the second natural language query into a second query language query (2604).

19. The computer-readable storage medium of claim 18, wherein the method further comprises: Receive a database embedding and a pair of embeddings, wherein the database embedding describes the context of a portion of the database, and the pair of embeddings describes the context of the dataset pair; as well as The predicted directory information is determined based on the similarity between the pair of embeddings and a subset of the database embeddings.

20. The computer-readable storage medium of claim 19, wherein the second prompt word is generated based on the first enhancement pair, and the method further comprises: Receive an indication of negative feedback for the query in the second query language; A correction pair is generated based on the instruction and the second natural language query. The correction pair includes the second natural language query and a corrected query language query, wherein the corrected query language query is a syntactically valid transformation of the second natural language query. Update the synthetic data to include the correction pair.

Images