Data intelligent interaction enhanced query method and system based on large model

Abstract

The invention discloses a data intelligent interaction enhancement query method and system based on a large model. The method comprises the steps that S1, a user inputs a query problem through a webpage interface; s2, determining a target data table in the candidate data tables, and obtaining table structure and attribute information; s3, the query problem is rewritten; s4, acquiring a question and SQL statement pair set of which the similarity ranks top k with the query question; s5, filling a first cue word template with the rewritten query problem, the table structure and attribute of the target data table, the problem and the SQL statement, and inputting the first cue word template into a large language model to generate a target query statement; s6, executing query in the corresponding data source according to the target query statement to obtain a target query result; and S7, filling the target query result and the query question into a second cue word template, inputting the target query result and the query question into the large language model, generating a query result in a natural language form, and returning the query result to the user. According to the invention, the performance and practicability of the interactive data query system are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of database query, and particularly to a data intelligent interaction enhanced query method and system based on a large model. Background Art

[0002] With the rapid development of information technology, databases have become important tools for enterprises and organizations to manage data. In practical applications, it is necessary to frequently perform query operations on databases to obtain the required information. Traditional database query methods mainly rely on Structured Query Language (SQL), which requires users to have professional programming knowledge and database operation skills. For users without a professional technical background, it is difficult to directly write SQL statements for data query. Although there are some visualization query tools on the market currently, these tools are often limited to fixed query templates and are restricted by the lack of understanding of query problems. They are neither suitable for natural language queries with very colloquial descriptions nor can they meet flexible and changeable query requirements. Summary of the Invention

[0003] In order to solve the technical problems existing in the prior art, such as the difficulty for the data query system to accurately understand natural language queries, poor readability of query results, and low query accuracy, the present invention provides a data intelligent interaction enhanced query method and system based on a large model. The specific solutions are as follows:

[0004] A large-scale intelligent interactive data query method includes the following steps:

[0005] S1, the user inputs a query problem through a web interface;

[0006] S2, determine the target data table in the candidate data tables based on the query problem, and obtain the table structure and attribute information of the target data table;

[0007] S3, rewrite the query problem by using a two-stage subject extraction algorithm based on regular matching and part-of-speech tagging and embedding model technology;

[0008] S4, use the embedding model technology to obtain a set of problems and SQL statement pairs with the top k similarities to the query problem;

[0009] S5, fill the rewritten query problem, the table structure and attributes of the target data table, and the problems and SQL statements into the first prompt template, and input them into the large language model to generate the target query statement;

[0010] S6, execute the query in the corresponding data source according to the target query statement to obtain the target query result;

[0011] S7, fill the target query result and the query question into the second prompt word template, input it into the large language model, generate the query result in natural language form, and return it to the user.

[0012] Preferably, the generation scope of the data query statement in step S2 includes but is not limited to MySQL, Oracle, PostgreSQL, Doris, and SQLite databases.

[0013] Preferably, determining the target data table in step S2 specifically includes the following steps:

[0014] S21, concatenating the table name and field name of the candidate data table according to the template to obtain the candidate data table information;

[0015] S22, inputting the query question and the candidate data table information into the embedding model respectively, obtaining respective vector representations, and calculating the cosine similarity between the query question vector and the candidate data table information vector;

[0016] S23, based on the cosine similarity score between the vectors, select a data table with a score exceeding a threshold as a target data table.

[0017] Preferably, the step S3 of rewriting the query question includes:

[0018] S31, extracting the subject of the query question based on a two-stage subject extraction algorithm to obtain the query subject name of the query question;

[0019] S32, obtaining all candidate subject names in the candidate data table based on a preset SQL query statement;

[0020] S33, performing offline vector representation of the candidate subject name based on the embedding model, and performing online vector representation of the query subject name based on the embedding model; using cosine similarity to calculate the similarity score between the two, obtaining the candidate subject name with the highest similarity that exceeds a preset threshold, and replacing the original query subject name with the candidate subject name to obtain a rewritten query question.

[0021] Preferably, the two-stage subject extraction algorithm in step S3 includes:

[0022] The first stage uses a regular matching method based on a preset regular expression to obtain a wide range of query subject names in the query question;

[0023] The second stage uses part-of-speech tagging natural language processing technology to remove non-name words from the query subject name, and performs post-processing operations such as splitting, merging, and deduplication to obtain a refined query subject name.

[0024] Preferably, step S4 specifically includes the following steps:

[0025] S41. Perform offline masking processing on the questions in the question and SQL statement pair set to obtain a masked question set;

[0026] S42. Based on the embedding model technology, offline construct a vector representation set of masked questions and online construct a vector representation of the query question;

[0027] S43. According to the vector representation of the query question and the vector representation set of masked questions, calculate the cosine similarity between the vector representations, and select the k masked questions with the highest similarity and the corresponding question and SQL statement pairs as example prompts for the large language model.

[0028] Preferably, the target query statement obtained in step S5 specifically includes:

[0029] If the user's query question is not relevant to the database, fill the query question into the first prompt word template, input the filled prompt word into the large language model, and the first prompt word controls the large language model to directly generate an answer for the corresponding general dialogue scenario, and then the system returns the answer content to the user interface;

[0030] If the user's query question is relevant to the database, fill the rewritten query question, the table structure and attributes of the target data table, and the example of the question and SQL statement pair into the first prompt word template, input the filled prompt word into the large language model, and obtain the target query statement.

[0031] Preferably, step S7 specifically includes: Based on the large language model technology, a corresponding SQL query statement is generated according to the user's query question. If the execution result of the SQL query statement is not empty, convert the execution result into Markdown format and fill it into the second prompt word template; if the execution result of the SQL query statement is empty, fill the information template indicating emptiness into the second prompt word template and input it into the large language model.

[0032] A system based on any of the above data intelligent interaction enhanced query methods based on large models includes:

[0033] A user interface for receiving the query question input by the user;

[0034] A target database matching module for determining the target data table;

[0035] A question rewriting module for rewriting the query question;

[0036] A question-SQL pair matching module for obtaining similar question and SQL statement pairs;

[0037] A query generation module for generating the target query statement;

[0038] A data query module for performing query operations;

[0039] A result processing module for generating natural language results.

[0040] The present invention also discloses a computer system, including a processor and a storage medium. The storage medium stores a computer program, and the processor reads and runs the computer program from the storage medium to execute the method described in any one of the above.

[0041] The beneficial effects of the present invention are as follows:

[0042] 1. By introducing large model technology, the accuracy of natural language queries is significantly improved;

[0043] 2. By adopting a two-stage entity extraction algorithm, the accuracy of query intention understanding is improved;

[0044] 3. By using a prompt template mechanism, the quality of query conversion and result generation is optimized;

[0045] 4. The natural language explanation of query results is realized, improving the usability of the system;

[0046] 5. It has good scalability and can adapt to query requirements in different fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0048] Figure 1 It is a schematic diagram of the system architecture for applying the data query method in the embodiment of the present invention;

[0049] Figure 2 It is a schematic diagram of the method flow according to the present invention;

[0050] Figure 3 It is a schematic diagram of the system flow according to the present invention;

[0051] Figure 4 It is a schematic diagram of the steps for determining the target data table according to the present invention;

[0052] Figure 5 It is a schematic diagram of the steps of the problem rewriting module in the embodiment provided by the present invention;

[0053] Figure 6It is a schematic diagram of the steps of the problem and SQL pair matching module according to the embodiments provided by the present invention. Detailed implementation manners

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0055] The present invention discloses a method and system for enhancing data intelligent interaction query based on a large model. Through large model technology, prompt word technology, and a two-stage entity extraction algorithm, the performance and practicality of the interactive data query system are significantly improved, and the comprehensive service ability of the system is enhanced.

[0056] As Figure 1 It is the system architecture for applying the data query method of the embodiments of the present invention.

[0057] The user is the direct user of the system and submits a query request to the system through natural language, such as asking for certain data or information.

[0058] The front-end interface is the window for the user to interact with the system. The user inputs a query request through the front-end interface, and the front-end interface passes the request to the Text2SQL service at the back end. After receiving the query result from the database, the result is displayed to the user.

[0059] The Text2SQL service is the core module of the system and is used to convert the natural language input by the user into a structured SQL query statement. The Text2SQL service receives the user query request from the front end, parses and generates the corresponding SQL query statement through natural language processing (NLP) technology, and then sends the query statement to the database.

[0060] The database stores the data that the system needs to query. The SQL query generated by the Text2SQL service is executed on the database. After the database returns the query result, it is fed back to the user via the Text2SQL service and the front-end interface.

[0061] As Figure 2 , the method flow steps of the present invention are as follows:

[0062] S1, the user inputs a query problem through the web interface.

[0063] S2, based on the query problem, determine the target data table from the candidate data table, and obtain the table structure and attribute information of the target data table; the generation scope of the data query statement includes but is not limited to MySQL, Oracle, PostgreSQL, Doris, and SQLite databases.

[0064] Specifically, Figure 4 As shown, determining the target data table specifically includes the following steps:

[0065] S21, concatenating the table name and field name of the candidate data table according to the template to obtain the candidate data table information;

[0066] S22, inputting the query question and the candidate data table information into the embedding model respectively, obtaining respective vector representations, and calculating the cosine similarity between the query question vector and the candidate data table information vector;

[0067] S23, based on the cosine similarity score between the vectors, select a data table with a score exceeding a threshold as a target data table.

[0068] S3, rewrites the query using a two-stage subject extraction algorithm based on regular matching and part-of-speech tagging and embedding model technology.

[0069] Specifically, Figure 5 As shown, the steps to rewrite the query include:

[0070] S31, extracting the subject of the query question based on a two-stage subject extraction algorithm to obtain the query subject name of the query question;

[0071] S32, obtaining all candidate subject names in the candidate data table based on a preset SQL query statement;

[0072] S33, performing offline vector representation of the candidate subject name based on the embedding model, and performing online vector representation of the query subject name based on the embedding model; using cosine similarity to calculate the similarity score between the two, obtaining the candidate subject name with the highest similarity that exceeds a preset threshold, and replacing the original query subject name with the candidate subject name to obtain a rewritten query question.

[0073] Among them, the two-stage subject extraction algorithm includes:

[0074] The first stage uses a regular matching method based on a preset regular expression to obtain a wide range of query subject names in the query question;

[0075] The second stage uses part-of-speech tagging natural language processing technology to remove non-name words from the query subject name, and performs post-processing operations such as splitting, merging, and deduplication to obtain a refined query subject name.

[0076] S4. Use the embedding model technology to obtain a set of the top k questions and SQL statement pairs with the highest similarity to the query question. As Figure 6 shown, it specifically includes the following steps:

[0077] S41. Perform offline masking processing on the questions in the set of question and SQL statement pairs to obtain a set of masked questions;

[0078] S42. Based on the embedding model technology, offline construct a set of vector representations of the masked questions and online construct a vector representation of the query question;

[0079] S43. According to the vector representation of the query question and the set of vector representations of the masked questions, calculate the cosine similarity between the vector representations, and select the question and SQL statement pairs corresponding to the top k masked questions with the highest similarity as few-shot examples for the large language model.

[0080] S5. Fill the rewritten query question, the table structure and attributes of the target data table, and the question and SQL statement pairs into the first prompt template, and input it into the large language model to generate the target query statement.

[0081] If the user's query question is not related to the database, fill the query question into the first prompt template, input the filled prompt into the large language model, and the first prompt controls the large language model to directly generate an answer for the corresponding general conversation scenario, and then the system returns the answer content to the user interface;

[0082] If the user's query question is related to the database, fill the rewritten query question, the table structure and attributes of the target data table, and the question and SQL statement pairs into the first prompt template, input the filled prompt into the large language model, and obtain the target query statement.

[0083] S6. Execute the query in the corresponding data source according to the target query statement to obtain the target query result.

[0084] S7. Fill the target query result and the query question into the second prompt template, input it into the large language model, generate the query result in natural language form, and return it to the user.

[0085] Based on the large language model technology, a corresponding SQL query statement is generated according to the user's query question. If the execution result of the SQL query statement is not empty, convert the execution result into Markdown format and fill it into the second prompt template; if the execution result of the SQL query statement is empty, fill the information template indicating emptiness into the second prompt template and input it into the large language model.

[0086] As Figure 3As shown, the system for the large model-based data intelligent interaction enhanced query method described above includes:

[0087] A user interface for receiving query questions input by the user; the user inputs a query question through the interface, and the system receives this natural language question and starts the processing flow.

[0088] A target database matching module for determining the target data table. The target database matching module means that the system further determines, through the target database matching module, the target data table related to the user's query question from the candidate database tables, and obtains the structure and attribute information of this table to provide data support for SQL generation.

[0089] A question rewriting module for rewriting the query question. The question rewriting module means that after the system receives the user's question, the system first reconstructs the question through the question rewriting module. This module uses natural language processing technologies such as regular matching, part-of-speech tagging, and embedding models to rewrite the user's query question into a structured expression more suitable for subsequent SQL generation.

[0090] A question-SQL pair matching module for obtaining similar question and SQL statement pairs. The question-SQL pair matching module means that the system simultaneously starts the question-SQL pair matching module, and through the embedding model technology, matches the user's question with the existing historical question and SQL pair set for similarity, and obtains the top k similar questions and SQL pairs to help generate SQL statements suitable for the current query requirements.

[0091] A query generation module for generating the target query statement.

[0092] A data query module for performing the query operation.

[0093] A result processing module for generating natural language results.

[0094] The first prompt word template means that the system inputs the rewritten query question, similar questions and SQL pairs, and target data table information into the preset first prompt word template to generate a prompt word for this query; the first prompt word is used to guide the large model to generate an SQL statement, which is input into the pre-trained large language model, and then the large language model generates the corresponding SQL statement according to the input content.

[0095] The implementation of the question classification prompt word means that the system determines whether the generated SQL statement meets the user's query requirements through the question classification prompt word implementation module. If so, the SQL statement is submitted to the database for query; if not, it returns to the question rewriting module for further reconstruction of the question or adjustment of the prompt word.

[0096] SQL execution and result acquisition refer to that when an SQL statement is generated and classified, the system executes the SQL query statement in the target database to obtain the SQL execution result.

[0097] The second prompt template is to input the query question and the SQL execution result into the prompt template to generate a prompt for the query result, so as to guide the large model to generate the final natural language answer.

[0098] In summary, the data intelligent interaction enhanced query method and system provided by the present invention realize the full-process automation from the user's input of a natural language question to the generation of an SQL query and then to the result return through question rewriting, SQL pair matching, prompt generation, and multiple rounds of processing of the large model, greatly simplifying the user's database query process and improving the accuracy and usability of the query.

[0099] The present invention also discloses a computer system, including a processor and a storage medium. The storage medium stores a computer program, and the processor reads and runs the computer program from the storage medium to execute the method described in any one of the above.

[0100] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

[0101] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented using a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0102] The steps of a method or algorithm described in connection with the embodiments disclosed in this specification can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read from, and write to, the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In an alternative, the processor and the storage medium can reside as discrete components in a user terminal.

[0103] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or code. The computer-readable medium includes both a computer storage medium and a communication medium including any medium that facilitates transfer of a computer program from one place to another. A storage medium may be any available medium that can be accessed by a computer. By way of example and not limitation, such a computer-readable medium can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a web site, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable medium.

[0104] The previous description of the present disclosure is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to the present disclosure will be apparent to those skilled in the art, and the general principles defined herein can be applied to other variations without departing from the spirit or scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0105] Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data intelligent interactive enhanced query method based on a large model, characterized in that: The following steps are involved: S1, the user enters the query question through the web interface; S2, determining a target data table from the candidate data tables based on the query question, and obtaining the table structure and attribute information of the target data table; S3, rewrites the query using a two-stage subject extraction algorithm based on regular matching and part-of-speech tagging and an embedding model technology; S4, using the embedding model technology to obtain the top k question and SQL statement pair sets with similarity to the query question; S5, filling the rewritten query question, the table structure and attributes of the target data table, the question and the SQL statement, and the example into the first prompt word template, and inputting them into the large language model to generate the target query statement; S6, executing a query in a corresponding data source according to the target query statement to obtain a target query result; S7, fill the target query result and the query question into the second prompt word template, input it into the large language model, generate the query result in natural language form, and return it to the user.

2. The method according to claim 1, characterized in that: The generation scope of the data query statement in step S2 includes but is not limited to MySQL, Oracle, PostgreSQL, Doris, and SQLite databases.

3. The method according to claim 1, characterized in that Determining the target data table in step S2 specifically includes the following steps: S21, concatenating the table name and field name of the candidate data table according to the template to obtain the candidate data table information; S22, inputting the query question and the candidate data table information into the embedding model respectively, obtaining respective vector representations, and calculating the cosine similarity between the query question vector and the candidate data table information vector; S23, based on the cosine similarity score between the vectors, a data table with a score exceeding a threshold is selected as a target data table.

4. The method according to claim 1, characterized in that: Step S3 of rewriting the query question includes: S31, extracting the subject of the query question based on a two-stage subject extraction algorithm to obtain the query subject name of the query question; S32, obtaining all candidate subject names in the candidate data table based on a preset SQL query statement; S33, performing offline vector representation of the candidate subject name based on the embedding model, and performing online vector representation of the query subject name based on the embedding model; using cosine similarity to calculate the similarity score between the two, obtaining the candidate subject name with the highest similarity that exceeds a preset threshold, and replacing the original query subject name with the candidate subject name to obtain a rewritten query question.

5. The method according to claim 1, characterized in that: The two-stage subject extraction algorithm in step S3 includes: The first stage uses a regular matching method based on a preset regular expression to obtain a wide range of query subject names in the query question; The second stage uses part-of-speech tagging natural language processing technology to remove non-name words from the query subject name, and performs post-processing operations such as splitting, merging, and deduplication to obtain a refined query subject name.

6. The method according to claim 1, characterized in that Step S4 specifically includes the following steps: S41, performing offline masking processing on the questions in the set of questions and SQL statements to obtain a masked question set; S42, based on the embedding model technology, constructs a set of vector representations of mask questions offline, and constructs vector representations of query questions online; S43, based on the vector representation of the query question and the vector representation set of the mask question, calculate the cosine similarity between the vector representations, and select the question and SQL statement pairs corresponding to the k mask questions with the highest similarity as example prompts of the large language model.

7. The method according to claim 1, characterized in that The target query statement obtained in step S5 specifically includes: If the user's query question is not relevant to the database, the query question is filled into the first prompt word template, and the filled prompt word is input into the large language model. The first prompt word controls the large language model to directly generate an answer for the corresponding general dialogue scenario, and then the system returns the answer content to the user interface; If the user's query question is related to the database, the rewritten query question, the table structure and attributes of the target data table, and the question and SQL statement pair examples are filled into the first prompt word template, and the filled prompt words are input into the large language model to obtain the target query statement.

8. The method according to claim 1, characterized in that: Step S7 specifically includes: generating a corresponding SQL query statement based on the user's query question based on the large language model technology; if the execution result of the SQL query statement is not empty, converting the execution result into Markdown format and filling it into the second prompt word template; if the execution result of the SQL query statement is empty, filling the second prompt word template with an empty information template and inputting it into the large language model.

9. A system based on a large-scale intelligent interactive data query method according to any one of claims 1 to 8, characterized in that: include: A user interface for receiving query questions input by a user; A target database matching module is used to determine a target data table; Question rewriting module, used to rewrite query questions; The question-SQL pair matching module is used to obtain similar question and SQL statement pairs; A query generation module, used to generate a target query statement; Data query module, used to perform query operations; The result processing module is used to generate natural language results.

10. A computer system, characterized in that: The method comprises a processor and a storage medium, wherein a computer program is stored in the storage medium, and the processor reads and runs the computer program from the storage medium to execute the method as claimed in any one of claims 1 to 8.

Images