Context learning-based database query generation method and system and storage medium

Abstract

The invention discloses a context learning-based database query generation method and system and a storage medium, and the method comprises the steps: extracting keywords and phrases from a natural language query, retrieving values similar to the keywords in a database, and returning values most related to the keywords and corresponding columns thereof, retrieving description information related to the keywords and the phrases in a database directory; performing mode selection aiming at the most relevant value of each keyword and the corresponding column, namely screening out the column irrelevant to the natural language query, further selecting a table necessary for generating the SQL query, and finally reducing the mode to a minimum column set necessary for generating the SQL query; and correcting logic and grammar errors of the method model through the database mode, the natural language query, the generated candidate SQL query and the execution result thereof. According to the method, the efficient SQL query statement is generated by intelligently understanding the data context and the query requirement.

Description

Technical Field

[0001] The present invention belongs to the technical field of database query language generation, and in particular relates to a database query generation method, system and storage medium based on context learning. Background Art

[0002] Traditional SQL query systems may have limitations in natural language understanding, user interaction, complex query processing, semantic ambiguity processing, database schema adaptability, performance optimization, and error handling. Therefore, they require more personnel and have higher usage costs. In contrast, the Text2SQL system has been specially optimized and designed in these aspects to provide more accurate and efficient query services.

[0003] CN118296034A discloses a NL2SQL method and system combining LLM and BERT. The method uses less database information, such as values, etc., which may limit the system's ability to process complex queries that require more comprehensive database information. CN117932028A discloses a question-answering system based on NL2SQL. If the input time entity format does not conform to the preset format, the system needs to convert it, which may increase the complexity of processing. CN116821168A discloses an improved NL2SQL method based on a generative large language model. The method only preprocesses the database table, and no additional processing is performed on the database description, etc. CN117112732A discloses an NL2SQL method based on LLM multi-table adaptive selection. The method relies on a constructed data knowledge base, which may limit the adaptability of the system when processing unknown or new fields. CN115408506A discloses a NL2SQL method of joint semantic parsing and semantic component matching. The method improves the accuracy of prediction through joint modeling, but this optimization may be targeted at specific tasks and may not be effective for other types of queries. CN117235108A discloses a NL2SQL generation method based on graph neural network. The graph neural network may require higher computing resources, especially when processing large-scale graphs, and the method relies on the construction of graph structure, which may limit the adaptability of the system when processing unstructured or semi-structured data.

[0004] In summary, current database query generation methods mainly involve reliance on specific data formats, demand for computing resources, processing capabilities for complex query logic, and system adaptability to new fields. Each system and method has its specific application scenarios and advantages, but in actual applications, it may need to be adjusted and optimized for specific business needs and data environments, and the performance needs to be improved. Summary of the invention

[0005] In view of the above problems, the present invention provides a database query generation method, system, electronic device and storage medium based on context learning, aiming to generate efficient SQL query statements by intelligently understanding data context and query requirements.

[0006] According to a first aspect of an embodiment of the present disclosure, a method for generating a database query based on context learning is provided, the method comprising the following steps:

[0007] Extract keywords and phrases from natural language queries, search for values ​​similar to the keywords in the database, return the most relevant values ​​and their corresponding columns for each keyword, and retrieve descriptive information related to the keywords and phrases in the database directory;

[0008] Selecting a pattern for the most relevant values ​​of each keyword and its corresponding columns, including: filtering out columns that are irrelevant to the natural language query, and further selecting tables necessary for generating SQL queries, and finally reducing the pattern to the minimum set of columns necessary for generating SQL queries;

[0009] Generate candidate SQL queries using the obtained schema, value and description information with a minimum column set;

[0010] Correct the logic and syntax errors of the method model through database schema, natural language queries, generated candidate SQL queries and their execution results.

[0011] In some specific embodiments, before searching for values ​​similar to the keyword in the database, a local sensitive hashing technique is used to index unique values ​​in the database, and when searching for values ​​similar to the keyword in the database, the value most similar to the keyword is found by querying the index.

[0012] In some specific embodiments, after finding the value most similar to the keyword by querying the index, the edit distance and semantic similarity between the value and the keyword are calculated to further optimize the search results.

[0013] In some specific embodiments, the method includes using embedded vectors to represent descriptive information in a database directory, storing the embedded vectors in a vector database, and utilizing a similarity measure between embedded vectors during a retrieval process to ensure the relevance between the retrieved descriptive information and a natural language query.

[0014] In some specific embodiments, the logic and syntax errors of the method model are corrected through the database model, natural language query, generated candidate SQL query and its execution results, specifically including:

[0015] Execute the initially generated candidate SQL query and obtain its result;

[0016] Based on the retrieved values ​​most relevant to the keywords and their corresponding columns, and the retrieved description information related to the keywords and phrases in the database directory, the logical consistency of the values ​​and their corresponding columns and the description information is analyzed to determine whether the candidate SQL query results meet the expectations of the original natural language query;

[0017] Based on the retrieved values ​​most relevant to the keywords and their corresponding columns, as well as the retrieved description information related to the keywords and phrases in the database directory, a semantic consistency analysis is performed to determine whether the candidate SQL query results are consistent with the retrieved values ​​and their corresponding columns and description information;

[0018] If logical or semantic inconsistencies are found, a series of predefined rules and heuristics will be automatically applied to correct the SQL query.

[0019] In some specific embodiments, the process of modifying the SQL query is iterative, and the query is re-executed after each modification, and the execution result is evaluated again until the generated query can produce correct and expected results.

[0020] In some specific embodiments, the method includes generating and modifying candidate SQL queries based on the DeepSeek Coder model, and introducing noise during the training process, wherein the noise includes columns and tables that are similar to the correct pattern in naming conventions and semantic attributes but are erroneous.

[0021] According to a second aspect of an embodiment of the present disclosure, a database query generation system based on context learning is provided, the system comprising:

[0022] Entity and context retrieval module, used to extract keywords and phrases from natural language queries, search for values ​​similar to the keywords in the database, return the most relevant values ​​and their corresponding columns for each keyword, and retrieve descriptive information related to the keywords and phrases in the database directory;

[0023] A pattern selection module is used to select a pattern for the most relevant value of each keyword and its corresponding column, including: filtering out columns that are irrelevant to the natural language query, and further selecting tables necessary for generating SQL queries, and finally reducing the pattern to the minimum set of columns necessary for generating SQL queries;

[0024] The query generation module is used to generate candidate SQL queries using the obtained schema, value and description information with the minimum column set; and correct the logic and syntax errors of the method model through the database schema, natural language query, generated candidate SQL query and its execution results.

[0025] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned context-learning-based database query generation method when executing the program.

[0026] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the above-mentioned database query generation method based on context learning are implemented.

[0027] The disclosed embodiments provide a method, system, electronic device and storage medium for generating database queries based on contextual learning, which ensures that the system can generate more accurate SQL by understanding and utilizing the problem context information, including database columns, database descriptions and column descriptions. The present invention adopts a phased processing method to decompose the problem into manageable subtasks, and each phase is optimized for a specific task. This modular design improves the maintainability and scalability of the system, and also allows each component to be improved and optimized independently; the method of the present invention uses local sensitive hashing and vector databases to quickly retrieve the most relevant information to the user query from a large amount of data. This method not only improves the retrieval speed, but also enhances the system's ability to process large data sets; the method of the present invention performs well on databases in multiple fields, whether it is medical, educational or other professional fields, it can generate accurate SQL queries. This cross-domain performance proves the versatility and adaptability of the system; the present invention improves the generation quality of candidate SQL queries by fine-tuning the DeepSeek Coder 34B model, CHESS, and the fine-tuning model better understands the requirements of specific tasks, thereby improving the accuracy of queries; after generating a preliminary SQL query, the present invention will perform logical consistency and semantic analysis, rather than simply letting the large model correct itself, to ensure that the query results meet expectations.

[0028] In summary, the present invention accurately retrieves entities and descriptions related to the problem from a large amount of data through keyword extraction and context retrieval. This process not only involves efficient retrieval of database values, but also includes in-depth analysis of the database directory to ensure the accuracy and relevance of the selected information. Through this meticulous retrieval, the method of the present invention can provide accurate data and context support for the subsequent query generation stage, thereby significantly improving the accuracy and efficiency of SQL queries. In addition, at the same time, the method of the present invention also includes intelligent screening of database patterns, by gradually narrowing down the tables and columns that may be related, and finally extracting a minimum and sufficient pattern subset, providing a solid foundation for generating accurate SQL queries. This ability to fully utilize context information is a major advantage of the method of the present invention when processing complex database queries.

[0029] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present invention and, together with the description, serve to explain the principles of the present invention.

[0031] Figure 1 is a flow chart of a method for generating a database query based on context learning in an embodiment of the present invention;

[0032] Figure 2 It is an overall architecture diagram of a database query generation method model based on context learning in an embodiment of the present invention;

[0033] Figure 3 This is a schematic diagram of a flow chart of a median selection process in an embodiment of the present invention;

[0034] Figure 4 is a selection flow chart based on semantics in an embodiment of the present invention;

[0035] Figure 5 Schematic diagram of a self-correction process in an embodiment of the present invention;

[0036] Figure 6 It is a schematic diagram of the structure of a database query generation system based on context learning in an embodiment of the present invention;

[0037] Figure 7 is a schematic diagram of an electronic device in an embodiment of the present invention;

[0038] Figure 8 This table shows the execution accuracy (EX) results of different large language model settings on the sampled BIRD dataset in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0040] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0041] The technical terms of the present invention are described as follows:

[0042] Large Language Models (LLMs): refers to large language models, such as GPT-4, that can understand and generate natural language text.

[0043] Text-to-SQL: A technique for converting natural language questions directly into SQL query statements to retrieve information from a database.

[0044] Schema: The database schema defines the structure of the database, including tables, columns, data types, primary keys, foreign keys, etc.

[0045] SQL queries: Structured Query Language queries are used to query and manipulate data in the database.

[0046] Entity and context retrieval: Entity and context retrieval refers to retrieving entity and context information related to user questions from the database.

[0047] Locality-Sensitive Hashing (LSH): An algorithm for approximate nearest neighbor search that can efficiently find the most similar items to a given query in large datasets.

[0048] Vector databases: Vector databases use the vector space model to store and retrieve data and are often used to process data that requires similarity comparison.

[0049] Ablation studies: A research method that removes or modifies parts of a model to evaluate the impact of those parts on the overall performance.

[0050] Execution accuracy: Execution accuracy refers to the accuracy with which the generated SQL query matches the correct query result when executed on the database.

[0051] BIRD dataset: A public dataset for evaluating text-to-SQL tasks, containing databases and question-SQL pairs from multiple professional fields.

[0052] Chain-of-thought (CoT) prompting: A prompting technique that guides the model to deeper reasoning by providing steps to solve a problem.

[0053] Zero-shot in-context learning: Zero-shot in-context learning refers to the model understanding and performing tasks through contextual information without additional training.

[0054] Fine-tuning: Fine-tuning a pre-trained model to adapt it to a specific task or dataset.

[0055] Spider dataset: Another dataset for evaluating text-to-SQL tasks, containing multiple databases and corresponding question-SQL pairs.

[0056] In today's data-driven era, the core of intelligent systems lies in efficiently processing and analyzing large amounts of data. However, the efficient use of data is almost completely limited by the ability of the SQL writer. As an advanced SQL generation tool, the present invention is designed to address this challenge. It optimizes the generation process of SQL queries in a context-aware manner, thereby achieving efficient use of data without sacrificing data privacy.

[0057] In addition, traditional database query systems play an important role in data management and query with their stability, maturity and powerful SQL functions. Nowadays, many query systems with their own characteristics have been born in combination with real-life scenarios and work needs. However, the effective use of these systems often relies on professionals who are proficient in SQL language. If such talents are relied on on a large scale, not only is the demand huge, but the cost is also high. At the same time, the existing natural language to SQL (NL2SQL) technology enables users to interact with the database in a more intuitive and friendly way by converting natural language queries into SQL statements. Although this technology significantly improves the user experience, it still has shortcomings in making full use of database information, training a large language model (LLM) with excellent results, and realizing an accurate and complete entity screening process. These limitations hinder the performance and application of NL2SQL technology in handling certain complex query scenarios.

[0058] The design concept of the present invention is to prioritize the efficiency and accuracy of data analysis while taking into account the security and privacy protection of data. It generates efficient SQL query statements by intelligently understanding the data context and query requirements, making the data analysis process both safe and efficient. This innovative method not only reduces the risk of data leakage, but also improves the performance of data processing, providing a powerful tool for data analysis and big data applications. The present invention provides a database query generation method, system, electronic device and storage medium based on context learning, with the following embodiments:

[0059] like Figure 1 As shown, a database query generation method based on context learning includes the following steps:

[0060] S1. Extract keywords and phrases from natural language queries, search for values ​​similar to the keywords in the database, return the most relevant values ​​and their corresponding columns for each keyword, and search for descriptive information related to the keywords and phrases in the database directory;

[0061] S2. Selecting a pattern for the most relevant value of each keyword and its corresponding column, including: filtering out columns irrelevant to the natural language query, and further selecting tables necessary for generating the SQL query, and finally reducing the pattern to the minimum column set necessary for generating the SQL query;

[0062] S3. Generate candidate SQL queries using the acquired schema, value and description information with the minimum column set; correct the logic and syntax errors of the method model through the database schema, natural language query, generated candidate SQL queries and their execution results.

[0063] The specific implementation process is as follows Figure 2 As shown in the figure, in the Entity and Context Retrieval stage, Keyword Extraction first extracts keywords and phrases from the user's question. This is done by providing the model with some question samples and asking it to identify and extract important vocabulary and entities; during Entity Retrieval, the extracted keywords are used to retrieve similar values ​​in the database, and the most relevant values ​​for each keyword and their corresponding columns are returned. This step uses the edit distance similarity measure and a hierarchical retrieval strategy based on Local Sensitive Hashing (LSH) to improve retrieval efficiency. During Context Retrieval, in addition to the value, relevant descriptive information in the database catalog is also retrieved. This descriptive information is crucial to understanding the entities and their relationships in the database, and can help the model generate SQL queries more accurately.

[0064] like Figure 2As shown in the figure, in the Schema Selection stage, during Individual Column Filtering, based on the top n most relevant values ​​found above, columns with little relevance to the problem are filtered out, and only the most relevant columns are retained; during Table Selection, after filtering out irrelevant columns, the tables necessary for generating SQL queries are further selected. This step requires the model to evaluate the relevance of each table and select only those tables that are critical for generating SQL queries; during Final Column Selection, in the final stage of schema selection, the schema is reduced to the minimum set of columns necessary to generate SQL queries. This step includes the model evaluating the necessity of each column and selecting the required columns.

[0065] like Figure 2 As shown in the figure, in the query generation phase, during candidate generation, after having the minimum and sufficient schema and relevant context, the model generates an SQL query that answers the user's question. The model uses the minimum schema, relevant values, and description information obtained in the previous step to generate candidate SQL queries. In the revision process, after generating candidate SQL queries, potential logical and syntactic errors are corrected by providing the database schema generated in the previous schema selection step, the question, the generated candidate SQL query, and its execution results. The model evaluates the correctness of the SQL query and revise it if necessary.

[0066] like Figure 3 As shown, before retrieving values ​​similar to keywords in the database, the local sensitive hashing technology is used to index the unique values ​​in the database. When retrieving values ​​similar to keywords in the database, the value most similar to the keyword is found by querying the index. Specifically, before entity retrieval, the local sensitive hashing (LSH) technology is used to index the unique values ​​in the database. LSH is an efficient technology suitable for approximate nearest neighbor search, which allows the system to quickly retrieve the values ​​most similar to a given keyword from a large amount of data. In the preprocessing stage, the system performs LSH indexing on the values ​​in the database, so that in the entity retrieval stage, the value most similar to the keyword can be quickly found by querying this index.

[0067] like Figure 3As shown, after finding the value most similar to the keyword by querying the index, the edit distance and semantic similarity between the value and the keyword are calculated to further optimize the retrieval results. Specifically, by combining the edit distance and semantic similarity, the method of the present invention not only relies on the exact match between the keyword and the database value, but also combines the edit distance and semantic similarity metrics to retrieve values. This method can handle variants in user questions, spelling errors, or inconsistencies in data storage formats. For each keyword, the system first uses LSH to retrieve the most similar values, and then calculates the edit distance and semantic similarity between these values ​​and the keyword to further optimize the retrieval results.

[0068] like Figure 4 As shown, the method also includes using embedded vectors to represent the descriptive information in the database directory, and storing the embedded vectors in the vector database. The similarity measure between the embedded vectors is used in the retrieval process to ensure the relevance between the retrieved descriptive information and the natural language query. Specifically, the method includes the creation of a vector database, and the descriptive information in the database directory, such as the description of the column, the description of the value, etc., is essential for understanding the database content and generating accurate SQL queries. The present invention preprocesses these descriptive information into a vector database so as to perform efficient semantic similarity search in the context retrieval stage. The system uses embedded vectors to represent each description and stores these vectors in the vector database; the method also includes semantic similarity retrieval, and in the context retrieval stage, the description that is most semantically relevant to the user's question is found by querying the vector database. This step utilizes the similarity measure between the embedded vectors to ensure that the retrieved descriptive information is highly relevant to the user's question. In this way, the most relevant contextual information can be provided to the model, helping the model to understand the problem more accurately and generate SQL queries.

[0069] Through the above preprocessing steps, the method of this embodiment can ensure that in the subsequent processing flow, the model can receive accurate and relevant data and context information, thereby improving the performance and accuracy of the entire system. This sophisticated preprocessing strategy is one of the key factors that enable the system to effectively handle complex database queries.

[0070] Correct the logic and syntax errors of the method model through database schema, natural language query, generated candidate SQL query and its execution results, including:

[0071] Execute the initially generated candidate SQL query and obtain its result;

[0072] Based on the retrieved values ​​most relevant to the keywords and their corresponding columns, and the retrieved description information related to the keywords and phrases in the database directory, the logical consistency of the values ​​and their corresponding columns and the description information is analyzed to determine whether the candidate SQL query results meet the expectations of the original natural language query;

[0073] Based on the retrieved values ​​most relevant to the keywords and their corresponding columns, as well as the retrieved description information related to the keywords and phrases in the database directory, a semantic consistency analysis is performed to determine whether the candidate SQL query results are consistent with the retrieved values ​​and their corresponding columns and description information;

[0074] If logical or semantic inconsistencies are found, a series of predefined rules and heuristics will be automatically applied to correct the SQL query.

[0075] In a preferred embodiment, the SQL query modification process is iterative, and the query is re-executed after each modification, and the execution result is re-evaluated until the generated query can produce correct and expected results.

[0076] Specifically, Figure 5 As shown in the figure, the context-based automatic correction mechanism is a key feature to ensure the accuracy of SQL queries. This mechanism works at the end of the query generation phase. After the preliminary SQL query is generated, the database schema, the original question, and the expected results of the query are used to evaluate and correct the generated SQL query. Specifically, the following steps are performed:

[0077] Execution result analysis: First, execute the SQL query generated initially and obtain its results.

[0078] Logical consistency check: Based on the results of entity and context retrieval, analyze the logical consistency of these results to determine whether they meet the expectations of the original query.

[0079] Semantic analysis: Based on the results of entity and context retrieval, the system further performs semantic analysis to ensure that the query results are consistent with the entity and description information in the question context.

[0080] Automatic Correction: A series of predefined rules and heuristics are automatically applied to correct the SQL query if any logical errors or inconsistencies are found.

[0081] Iterative Optimization: The correction process may be iterative, where the query is re-executed after each correction and the results are evaluated again until the resulting query produces correct and expected results.

[0082] This context-based automatic correction mechanism not only improves the accuracy of SQL queries, but also enhances the robustness of the method, enabling it to better handle complex queries and changing data environments. The use of this mechanism significantly improves the reliability and efficiency of the conversion from natural language to SQL queries.

[0083] The method embodiment also includes generating and correcting candidate SQL queries based on the DeepSeek Coder model, and introducing noise during the training process, wherein the noise includes columns and tables that are similar to the correct pattern in naming conventions and semantic attributes but are erroneous.

[0084] Specifically, the embodiment uses the fine-tuned DeepSeek Coder 34B model with excellent performance, and its training process includes:

[0085] Dataset preparation:

[0086] Fine-tuning is performed using a training set of approximately 9,500 samples from the BIRD benchmark.

[0087] To address the error propagation problem, some noise is introduced when constructing the dataset, including some columns and tables that are similar to the correct schema in naming conventions and semantic properties but are erroneous.

[0088] Model selection: The DeepSeek Coder 34B model is selected as the base model for fine-tuning. This model is a large language model capable of handling complex text-to-SQL tasks.

[0089] Fine-tuning techniques: Fine-tuning using a 4-bit quantized base model and a LoRa adapter, a technique called QLoRa, enables efficient tuning of the model’s parameters to suit specific tasks.

[0090] Hyperparameter settings: The rank parameter of Lora is set to 128, the alpha parameter of Lora is set to 256, the batch size is 32, and the learning rate is 1e-4.

[0091] Training time: 4 hours on one H100 GPU.

[0092] Another embodiment is used to illustrate a database query generation system based on context learning, such as Figure 6 As shown, the system 600 includes:

[0093] Entity and context retrieval module 610, for extracting keywords and phrases from natural language queries, searching for values ​​similar to the keywords in the database, returning the most relevant values ​​and their corresponding columns for each keyword, and retrieving descriptive information related to the keywords and phrases in the database directory;

[0094] The mode selection module 620 is used to select a mode for the most relevant value of each keyword and its corresponding column, including: filtering out columns that are irrelevant to the natural language query, and further selecting tables necessary for generating the SQL query, and finally reducing the mode to the minimum set of columns necessary for generating the SQL query;

[0095] The query generation module 630 is used to generate candidate SQL queries using the acquired schema, value and description information with the minimum column set; and correct the logic and syntax errors of the method model through the database schema, natural language query, generated candidate SQL query and its execution results.

[0096] In addition to the above modules, the system 600 may also include other components. However, since these components are irrelevant to the content of the embodiment of the present disclosure, their illustration and description are omitted here.

[0097] The other specific working processes of the database query generation system 600 based on context learning refer to the description of the above-mentioned database query generation method embodiment based on context learning, which will not be repeated here.

[0098] Another embodiment is used to illustrate that the system of the present invention can also be used with the help of Figure 7 The architecture of the computing device shown is implemented. Figure 7 The architecture of the computing device is shown. Figure 7 As shown, a computer system 710, a system bus 730, one or more CPUs 740, an input / output 720, a memory 750, etc. The memory 750 can store various data or files used for computer processing and / or communication and program instructions executed by the CPU including the database query generation method based on context learning of the embodiment. Figure 7 The architecture shown is only exemplary and can be adjusted according to actual needs when implementing different devices. Figure 7 One or more components in. The memory 750, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the database query generation method based on context learning in the embodiment of the present invention (for example, the entity and context retrieval module 610, the mode selection module 620 and the query generation module 630 in the database query generation system 600 based on context learning). One or more CPUs 740 execute various functional applications and data processing of the system of the present invention by running the software programs, instructions and modules stored in the memory 750, that is, to implement the above-mentioned database query generation method based on context learning, and the method includes the following steps:

[0099] Extract keywords and phrases from natural language queries, search for values ​​similar to the keywords in the database, return the most relevant values ​​and their corresponding columns for each keyword, and retrieve descriptive information related to the keywords and phrases in the database directory;

[0100] Selecting a pattern for the most relevant values ​​of each keyword and its corresponding columns, including: filtering out columns that are irrelevant to the natural language query, and further selecting tables necessary for generating SQL queries, and finally reducing the pattern to the minimum set of columns necessary for generating SQL queries;

[0101] Generate candidate SQL queries using the obtained schema, value and description information with a minimum column set;

[0102] Correct the logic and syntax errors of the method model through database schema, natural language queries, generated candidate SQL queries and their execution results.

[0103] Of course, the processor of the server provided in the embodiment of the present invention is not limited to executing the method operations described above, but can also execute related operations in the context learning-based database query generation method provided in any embodiment of the present invention.

[0104] The memory 750 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 750 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 750 may further include a memory remotely arranged relative to one or more CPUs 740, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0105] The input / output 720 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The input / output 720 may also include a display device such as a display screen.

[0106] The embodiment of the present invention also provides a non-temporary computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by the processor, the database query generation method based on context learning described in the above embodiment is implemented. The computer-readable storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media (non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0107] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0108] The program code contained on the storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the foregoing.

[0109] In addition, other specific working processes of a non-temporary computer-readable storage medium refer to the description of the above-mentioned database query generation method embodiment based on context learning, and will not be repeated here.

[0110] In order to prove the effectiveness of the method, system, electronic device and storage medium of the present invention, a verification test was conducted. The experimental setting was that each experiment was divided into a triple (column filtering, candidate query generation, table / column selection + revision), and a large language model was used for each experiment. The test data set used the BIRD data set, and the test results are as follows: Figure 8As shown, it can be seen that the execution accuracy of the fine-tuned DeepSeek Coder model in the system has been significantly improved, especially the ability of the fine-tuned DeepSeek Coder model in understanding the problem context and database schema has been enhanced, which enables it to more accurately capture semantic information when generating SQL queries. This means that the large model fine-tuned by the present invention can surpass the closed-source GPT model in specific fields; the fine-tuned model shows better robustness when facing complex or ambiguous natural language queries. It can more accurately parse the user's intentions and generate correct SQL queries even when faced with ambiguous or ambiguous questions. The fine-tuned model has shown good performance on databases in different fields, which shows that the model can adapt to the specific language and structure of different fields through fine-tuning, thereby improving its cross-domain applicability. As Figure 8 , each stage of the last row uses an open source large model, and for the first time, an execution accuracy of 59.86% is achieved while ensuring privacy (without using a closed source large model).

[0111] Based on the technical solutions provided by the above embodiments, a method, system, electronic device and storage medium for generating database queries based on contextual learning decompose complex natural language processing tasks into smaller and more manageable subtasks through a multi-stage processing flow. This process includes three main stages: entity and context retrieval, mode selection and SQL generation. In the entity and context retrieval stage, advanced technologies such as local sensitive hashing (LSH) index and vector database are used to efficiently retrieve entity and context information related to user queries from a large amount of data. The innovation of this stage is that it can understand the user's query intent and accurately identify the data most relevant to the query. In the mode selection stage, the system further screens out the most relevant database mode to lay the foundation for generating accurate SQL queries. Finally, in the SQL generation stage, the system uses the fine-tuned DeepSeek Coder model to generate candidate SQL queries and makes necessary revisions to ensure the accuracy and efficiency of the query.

[0112] Another key innovation of the present invention is its adaptive schema pruning technology. When processing complex database queries, it is necessary to filter out a minimum and sufficient subset from a huge database schema that may contain hundreds of columns and multiple tables. The present invention uses adaptive schema pruning technology to dynamically adjust according to the complexity of the problem and the size of the model context to extract a minimum and sufficient subset of the database schema. This technology not only improves the efficiency of query generation, but also ensures the accuracy of the generated SQL queries. In this way, the method of the present invention can effectively process complex database queries while maintaining data privacy.

[0113] Another important innovation of the present invention is its fine-tuned DeepSeek Coder model. This model introduces noise during the training process to simulate the errors and uncertainties that may be encountered in real applications. In this way, the fine-tuned DeepSeek Coder model can better handle the complexity and diversity in real applications. In addition, a novel training dataset construction method is adopted to reduce error propagation through noise injection, further improving the robustness and accuracy of the model. This innovation enables more accurate understanding and processing of user query intent when generating SQL queries.

[0114] In addition, privacy protection is a core consideration. With the strengthening of data privacy regulations and the improvement of personal privacy awareness, the method of the present invention emphasizes the protection of sensitive information during data processing and query generation. By executing the query generation process locally on the client, the original data or database model is avoided from being sent to a third-party service provider, thereby reducing the risk of data leakage. In addition, the method is designed to minimize data exposure, providing the necessary information to the model only when necessary, and anonymizing or de-identifying the data during the processing process. This strict protection of privacy not only meets current legal requirements, but also enhances the user's trust in the system, making the method of the present invention a solution that provides efficient data analysis capabilities while protecting privacy. In this way, it is ensured that users can enjoy efficient data analysis without having to worry about their data security and privacy leakage.

[0115] The method of the present invention achieves new best performance on the cross-domain BIRD dataset, an achievement that demonstrates its wide applicability and efficiency in processing database queries in different fields. The BIRD dataset covers multiple professional fields, including medical, education, blockchain, and sports, which makes the cross-domain performance of the method of the present invention an important innovation. By training and testing on datasets from these different fields, it has been proven that it can understand and process various complex database queries, regardless of the field. This innovation makes the method of the present invention a truly universal text-to-SQL query generation tool that can meet the needs of different users.

[0116] In this document, the terms "comprises," "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, such that a step or method that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such step or method.

[0117] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A method for generating database queries based on context learning, characterized in that: The method comprises the following steps: Extract keywords and phrases from natural language queries, search for values ​​similar to the keywords in the database, return the most relevant values ​​and their corresponding columns for each keyword, and retrieve descriptive information related to the keywords and phrases in the database directory; Selecting a pattern for the most relevant values ​​of each keyword and its corresponding columns, including: filtering out columns that are irrelevant to the natural language query, and further selecting tables necessary for generating SQL queries, and finally reducing the pattern to the minimum set of columns necessary for generating SQL queries; Generate candidate SQL queries using the obtained schema, value and description information with a minimum column set; Correct the logic and syntax errors of the method model through database schema, natural language queries, generated candidate SQL queries and their execution results.

2. The method for generating database queries based on context learning according to claim 1, characterized in that: Before searching for values ​​similar to the keyword in the database, the local sensitive hashing technology is used to index the unique values ​​in the database. When searching for values ​​similar to the keyword in the database, the value most similar to the keyword is found by querying the index.

3. The method for generating database queries based on context learning according to claim 2, characterized in that: After finding the value most similar to the keyword by querying the index, the edit distance and semantic similarity between the value and the keyword are calculated to further optimize the retrieval results.

4. The method for generating database queries based on context learning according to claim 1, characterized in that: The method includes using embedded vectors to represent description information in a database directory, storing the embedded vectors in a vector database, and utilizing similarity metrics between embedded vectors during a retrieval process to ensure the relevance between the retrieved description information and a natural language query.

5. The method for generating database queries based on context learning according to claim 1, characterized in that: Correct the logic and syntax errors of the method model through database schema, natural language query, generated candidate SQL query and its execution results, including: Execute the initially generated candidate SQL query and obtain its result; Based on the retrieved values ​​most relevant to the keywords and their corresponding columns, and the retrieved description information related to the keywords and phrases in the database directory, the logical consistency of the values ​​and their corresponding columns and the description information is analyzed to determine whether the candidate SQL query results meet the expectations of the original natural language query; Based on the retrieved values ​​most relevant to the keywords and their corresponding columns, as well as the retrieved description information related to the keywords and phrases in the database directory, a semantic consistency analysis is performed to determine whether the candidate SQL query results are consistent with the retrieved values ​​and their corresponding columns and description information; If logical or semantic inconsistencies are found, a series of predefined rules and heuristics will be automatically applied to correct the SQL query.

6. The method for generating database queries based on context learning according to claim 5, characterized in that: The SQL query correction process is iterative. After each correction, the query is re-executed and the execution results are evaluated again until the generated query can produce correct and expected results.

7. The method for generating database queries based on context learning according to claim 1, characterized in that: The method comprises generating candidate SQL queries based on a DeepSeek Coder model and performing corrections, and introducing noise in a training process, wherein the noise comprises columns and tables that are similar to correct patterns in naming conventions and semantic attributes but are erroneous.

8. A database query generation system based on context learning, characterized in that: The system comprises: Entity and context retrieval module, used to extract keywords and phrases from natural language queries, search for values ​​similar to the keywords in the database, return the most relevant values ​​and their corresponding columns for each keyword, and retrieve descriptive information related to the keywords and phrases in the database directory; A pattern selection module is used to select a pattern for the most relevant value of each keyword and its corresponding column, including: filtering out columns that are irrelevant to the natural language query, and further selecting tables necessary for generating SQL queries, and finally reducing the pattern to the minimum set of columns necessary for generating SQL queries; The query generation module is used to generate candidate SQL queries using the obtained schema, value and description information with the minimum column set; and correct the logic and syntax errors of the method model through the database schema, natural language query, generated candidate SQL query and its execution results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for generating a database query based on context learning as described in any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instructions are executed by a processor, the steps of the method for generating a database query based on context learning as described in any one of claims 1 to 7 are implemented.

Images