Database query training method and device based on RAG and large model

Abstract

The invention provides a database query training method and device based on RAG and a large model, and the method comprises the steps: obtaining DDL annotation data and SQL annotation data, standardizing the data into a training document, vectorizing the training document into a retrieval knowledge vector, and storing the retrieval knowledge vector into a preset vector database; in response to a query operation of a user, vectorizing the query operation into a query vector, and retrieving matched associated DDL and associated SQL in a preset vector database according to the query vector; constructing a model cue word according to a query problem corresponding to the query operation, the associated DDL and the associated SQL, inputting the model cue word into a preset large language model, and converting the query problem into a corresponding target SQL; and determining a query result of the target SQL in the target business database, and feeding back the query result to the user. According to the method, intelligent conversion from the natural language to the SQL can be realized, and the vector retrieval is introduced to enhance the generation process, so that the illusion problem of a large model is remarkably reduced, and the accuracy and safety of generating the SQL are improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of database query, and in particular, to a database query training method and device based on RAG and a large model. Background Art

[0002] As enterprises continue to deepen their digital transformation, database query has become an indispensable key link in their daily operations. However, enterprise database platforms are usually composed of multiple database types (such as MySQL, MongoDB, Redis, etc.), and employees need to master specific query languages ​​(such as SQL) when interacting with these databases. This undoubtedly increases the usage threshold and learning cost for non-technical personnel, which in turn affects work efficiency. In order to solve this problem, database query technology based on natural language processing (NLP) has gradually attracted attention in recent years.

[0003] Traditional database query platforms attempt to achieve the conversion of natural language to SQL through Text2SQL technology. This method usually parses the semantics of natural language based on NLP models (such as Seq2Seq, BERT, etc.) and concatenates and generates SQL statements. However, due to the limited understanding of semantics by traditional models, the generated SQL needs to be labeled offline and repeatedly trained for each database type to achieve a high degree of accuracy. This not only consumes a lot of resources, but also makes it difficult to adapt to the dynamic changes in enterprise databases. In addition, traditional methods have limited support for complex queries, which easily leads to a high error rate in generating SQL, further limiting the widespread application of the technology. Summary of the invention

[0004] The disclosed embodiments at least provide a database query training method and device based on RAG and a large model, which can realize the intelligent conversion from natural language to SQL, and introduce a vector retrieval enhancement generation process, which significantly reduces the hallucination problem of the large model and improves the accuracy and security of the generated SQL.

[0005] The present disclosure provides a database query training method based on RAG and a large model, including:

[0006] Obtain DDL annotation data and SQL annotation data and standardize them into training documents, and vectorize the training documents into retrieval knowledge vectors and store them in a preset vector database;

[0007] In response to a user's query operation, vectorize the query operation into a query vector, and retrieve matching associated DDL and associated SQL in the preset vector database according to the query vector;

[0008] Construct a model prompt word according to the query question corresponding to the query operation, the associated DDL and the associated SQL, input the model prompt word into a preset large language model, and convert the query question into a corresponding target SQL;

[0009] Determine the query result of the target SQL in the target business database, and feed back the query result to the user.

[0010] In an optional implementation, receiving DDL annotation data and SQL annotation data input by a user and standardizing them into training documents specifically includes:

[0011] Receiving the DDL annotation data and the SQL annotation data input by the user through a preset online interactive interface;

[0012] Performing syntax checks on the DDL annotation data and the SQL annotation data respectively, wherein verifying whether the DDL annotation data contains complete annotation information of tables and fields, and verifying the level of detail of the description information corresponding to the SQL annotation data;

[0013] The DDL annotation data and the SQL annotation data are respectively standardized into corresponding target document formats to generate the training document.

[0014] In an optional implementation, vectorizing the training document into a retrieval knowledge vector and storing it in a preset vector database specifically includes:

[0015] Processing the training document through a preset vectorization algorithm model, and vectorizing the DDL annotation data and the SQL annotation data into retrieval knowledge vectors respectively;

[0016] The training document is stored in the corresponding table in the preset vector database according to the DDL data and the SQL data respectively.

[0017] In an optional implementation, in response to a user's query operation, the query operation is vectorized into a query vector, and matching associated DDL and associated SQL are retrieved from the preset vector database according to the query vector, specifically including:

[0018] Determine the query question corresponding to the query operation;

[0019] Vectorizing the query question into the query vector, and searching the DDL table and the SQL table included in the preset vector database according to the query vector;

[0020] A plurality of the associated DDLs and the associated SQLs whose similarity with the query vector is greater than a preset similarity threshold are determined.

[0021] In an optional implementation, constructing a model prompt word according to the query question corresponding to the query operation, the associated DDL, and the associated SQL specifically includes:

[0022] Inserting the query question into a preset basic prompt word template to generate a basic prompt word;

[0023] Inserting the associated DDL into a preset DDL reference prompt word template to generate a DDL prompt word, wherein the DDL prompt word indicates that the associated DDL is used as a reference for an answer corresponding to the query question;

[0024] Inserting the associated SQL into a preset SQL reference prompt word template to generate an SQL prompt word, wherein the SQL prompt word indicates that the associated SQL is used as a reference for an answer corresponding to the query question;

[0025] The basic prompt word, the DDL prompt word, and the SQL prompt word are combined to generate the model prompt word.

[0026] In an optional implementation manner, determining the query result of the target SQL in the target business database specifically includes:

[0027] Parsing the query attribute information included in the target SQL;

[0028] Performing a query operation in the target business database according to the query attribute information to determine the query result;

[0029] The query results are desensitized according to preset sensitive data rules.

[0030] In an optional implementation manner, after inputting the model prompt word into a preset large language model and converting the query question into a corresponding target SQL, the method further includes:

[0031] Determine the target table information included in the query attribute information;

[0032] Determining whether the user has the authority to query the target table information according to the target table information and the query authority information corresponding to the user;

[0033] If the user does not have the authority to query the target table information, authority exception information is returned to the user.

[0034] The disclosed embodiment also provides a database query training device based on RAG and a large model, including:

[0035] A training module is used to obtain DDL annotation data and SQL annotation data and standardize them into training documents, and vectorize the training documents into retrieval knowledge vectors and store them in a preset vector database;

[0036] An associated query module, used to vectorize the query operation into a query vector in response to a user's query operation, and retrieve matching associated DDL and associated SQL in the preset vector database according to the query vector;

[0037] A prompt word assembly module, used to construct a model prompt word according to the query question corresponding to the query operation, the associated DDL and the associated SQL, and input the model prompt word into a preset large language model to generate a target SQL;

[0038] The result query module is used to determine the query result of the target SQL in the target business database and feed back the query result to the user.

[0039] An embodiment of the present disclosure also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the above-mentioned database query training method based on RAG and large models, or steps in any possible implementation of the above-mentioned database query training method based on RAG and large models are performed.

[0040] The embodiments of the present disclosure also provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program executes the above-mentioned database query training method based on RAG and a large model, or the steps in any possible implementation of the above-mentioned database query training method based on RAG and a large model.

[0041] The embodiments of the present disclosure also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the above-mentioned database query training method based on RAG and a large model, or the steps in any possible implementation of the above-mentioned database query training method based on RAG and a large model.

[0042] The database query training method and device based on RAG and large model provided by the embodiment of the present disclosure obtains DDL annotation data and SQL annotation data and standardizes them into training documents, vectorizes the training documents into retrieval knowledge vectors and stores them in a preset vector database; in response to the user's query operation, vectorizes the query operation into a query vector, and retrieves matching associated DDL and associated SQL in the preset vector database according to the query vector; constructs model prompt words according to the query question corresponding to the query operation, the associated DDL and the associated SQL, inputs the model prompt words into the preset large language model, converts the query question into the corresponding target SQL; determines the query result of the target SQL in the target business database, and feeds back the query result to the user. Intelligent conversion from natural language to SQL can be realized, and a vector retrieval enhancement generation process is introduced, which significantly reduces the hallucination problem of the large model and improves the accuracy and security of the generated SQL.

[0043] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0045] Figure 1 A flowchart of a database query training method based on RAG and a large model provided in an embodiment of the present disclosure is shown;

[0046] Figure 2 A flowchart of another database query training method based on RAG and a large model provided in an embodiment of the present disclosure is shown;

[0047] Figure 3 A schematic diagram of a database query training device based on RAG and a large model provided by an embodiment of the present disclosure is shown;

[0048] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0050] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0051] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0052] Research has found that traditional database query platforms attempt to achieve the conversion of natural language to SQL through Text2SQL technology. This method usually parses the semantics of natural language based on NLP models (such as Seq2Seq, BERT, etc.) and concatenates and generates SQL statements. However, due to the limited understanding of semantics by traditional models, the generated SQL needs to be labeled offline and repeatedly trained for each database type to achieve a high degree of accuracy. This not only consumes a lot of resources, but also makes it difficult to adapt to the dynamic changes in enterprise databases. In addition, traditional methods have limited support for complex queries, which easily leads to a high error rate in generating SQL, further limiting the widespread application of the technology.

[0053] Based on the above research, the present disclosure provides a database query training method and device based on RAG and a large model, which obtains DDL annotation data and SQL annotation data and standardizes them into training documents, vectorizes the training documents into retrieval knowledge vectors and stores them in a preset vector database; in response to a user's query operation, vectorizes the query operation into a query vector, and retrieves matching associated DDL and associated SQL in the preset vector database according to the query vector; constructs model prompts according to the query question corresponding to the query operation, the associated DDL, and the associated SQL, inputs the model prompts into a preset large language model, and converts the query question into the corresponding target SQL; determines the query result of the target SQL in the target business database, and feeds back the query result to the user. Intelligent conversion from natural language to SQL can be achieved, and a vector retrieval enhancement generation process is introduced, which significantly reduces the hallucination problem of the large model and improves the accuracy and security of the generated SQL.

[0054] To facilitate understanding of the present embodiment, a database query training method based on RAG and a large model disclosed in the present embodiment is first introduced in detail. The execution subject of the database query training method based on RAG and a large model provided in the present embodiment is generally a computer device with a certain computing capability, and the computer device includes, for example: a terminal device or a server or other processing device, and the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the database query training method based on RAG and a large model can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0055] See also Figure 1 As shown, it is a flowchart of a database query training method based on RAG and a large model provided by an embodiment of the present disclosure, and the method includes steps S101 to S104, wherein:

[0056] S101, obtaining DDL annotation data and SQL annotation data and standardizing them into training documents, and vectorizing the training documents into retrieval knowledge vectors and storing them in a preset vector database.

[0057] In the specific implementation, DDL annotated data and SQL annotated data are the core training data sources of this solution. DDL mainly defines the table structure, fields and their constraints of the database, and provides semantic annotations and business background information for tables and fields through annotations to help understand their functions and uses when retrieving and matching queries. SQL is used to query, update, insert or delete data in the database. The purpose of annotating SQL is to provide a detailed functional description and business scenario explanation for each SQL statement so that the model can accurately understand the intent of the query operation.

[0058] Here, the annotation content for DDL can include table-level description, field-level description and constraint description. Table-level description describes the function and purpose of the entire table, for example, table users means "storing user information in the system, including user ID and user name"; field-level description adds detailed semantic annotations to each field in the table, explaining the meaning, purpose and constraints of the field, for example, field user_id: means "unique identifier of the user", field user_name: means "name of the user", etc.; constraint description annotates the constraints, for example, primary key constraint PRIMARY KEY(user_id): means "guarantee that the user ID is unique in the table and cannot be empty", and foreign key constraint describes the relationship between the field and other tables.

[0059] Here, the annotation content for SQL can include query purpose, business scenario and parameter explanation. The query purpose indicates the specific purpose of the SQL statement execution, such as SELECT user_id, user_name FROM users; used to "query the user ID and user name of all users"; the business scenario describes the business background and purpose of the SQL statement application, such as scenario description: "query the basic information of all registered users for export."; the parameter explanation explains the role of the parameters in detail for the dynamic parameters or conditions contained in the SQL statement, such as WHERE user_id=? means "query the user information corresponding to the specified user ID".

[0060] The annotated data provides a clear semantic background for the model, enabling it to better understand and generate SQL queries. Through annotation, the system can find more relevant DDL and SQL based on semantics during vector retrieval. It also helps the large language model to refer to relevant DDL and SQL information when generating SQL.

[0061] As a possible implementation method, DDL annotation data and SQL annotation data input by a user are received through a preset online interactive interface; syntax checking is performed on the DDL annotation data and the SQL annotation data respectively, wherein it is verified whether the DDL annotation data contains complete annotation information of tables and fields, and the detail level of the corresponding description information of the SQL annotation data is verified; the DDL annotation data and the SQL annotation data are standardized into corresponding target document formats respectively to generate training documents.

[0062] Specifically, the DDL and SQL annotation data entered by the user are obtained through an online interactive interface. After that, the syntax of the DDL statement is ensured to be correct, whether it contains complete table description and field annotation information, and the DDL content is verified one by one through open source parsing tools (such as MySQL's syntax tree parsing engine). Verify the syntax correctness of the SQL statement, ensure that the query logic is correct, and ensure that each SQL statement is accompanied by a detailed business description.

[0063] Here, the DDL annotation data and SQL annotation data input by the user are converted into a unified target document format to ensure that each record in the document contains grammatically correct DDL or SQL primitives and their corresponding detailed description information.

[0064] For example, the training document format corresponding to the DDL annotation data may be: creat table xxx(id intpromay key comment'unique key',name varchar(150)null cpmment'name'); the training document format corresponding to the SQL annotation data may be: {"sql":"select*forom xxx","desc":"query xxx table"}.

[0065] Furthermore, the training documents are processed by a preset vectorization algorithm model, and the DDL annotation data and the SQL annotation data are respectively vectorized into retrieval knowledge vectors; the training documents are stored in corresponding tables in a preset vector database according to the DDL data and the SQL data.

[0066] Here, in order to support subsequent query operations, the training documents need to be vectorized and stored in the vector database. First, a vectorization algorithm model is selected to encode the document description (DDL description or SQL description) into a high-dimensional vector to retain the semantic features. For example, the vectorization algorithm model can be all-MiniLM-L6-v2, sentence-transformers, etc. The description of each training document (such as the desc field) is input into the vectorization model, and the structural features of the SQL or DDL statement (such as field name, table name) are encoded as auxiliary vectors.

[0067] The vectorized data is stored using a vector database (such as Pinecone, Milvus, Weaviate, etc.), the vectors of DDL labeled data are stored in the DDL table, and the vectors of SQL labeled data are stored in the SQL table.

[0068] In this way, standardized and grammatically correct training documents can be provided to improve the vector quality of the vectorized model, and the vectorized data can be stored in a clearly structured vector database to quickly query and retrieve the associated DDL and SQL content, providing high-quality input for subsequent model prompt word construction.

[0069] S102: In response to a query operation of a user, vectorize the query operation into a query vector, and retrieve matching associated DDL and associated SQL in the preset vector database according to the query vector.

[0070] In the specific implementation, the query problem is vectorized into a query vector, and the DDL table and SQL table included in the preset vector database are searched according to the query vector; multiple associated DDL and associated SQL with similarity to the query vector greater than a preset similarity threshold are determined. The user's query operation is converted into a vector form that can be understood and calculated by the model, and the DDL and SQL related to the query operation semantics are found from the preset vector database through vector retrieval technology.

[0071] Here, the user query operation may be a natural language question, a keyword combination, or a context-based query request, such as a natural language question: "Query the names and ages of all users in the user table.", keyword: "user name query".

[0072] Among them, vectorization is the process of converting the user's query operation (in text form) into a multidimensional numerical vector so that it can be matched with the vector in the database. This algorithm is consistent with the vectorization algorithm in the training process, and both can use the all-MiniLM-L6-v2 model.

[0073] Further, a query problem corresponding to the query operation is determined; the query problem is vectorized into a query vector, and a search is performed in a DDL table and an SQL table included in a preset vector database according to the query vector; and multiple associated DDLs and associated SQLs whose similarity to the query vector is greater than a preset similarity threshold are determined.

[0074] In the specific implementation, in the preliminary preparation, the DDL annotation data and SQL annotation data have been vectorized into index vectors and stored in the vector database. Each vector corresponds to a specific DDL or SQL statement and is accompanied by annotation information. The cosine similarity or Euclidean distance is used to compare the similarity between the user query vector and the vector stored in the database. Vectors with similarity higher than the preset threshold are found. These DDL or SQL are considered to be semantically related to the user query and multiple candidate results are returned.

[0075] Here, the search results are sorted according to the similarity scores, and the most relevant DDL and SQL are returned first. Each returned result includes the matching DDL or SQL statement and its annotation information. The vector search process can use the approximate nearest neighbor search algorithm (ANNS).

[0076] In this way, through vectorization and database retrieval, semantically related DDL and SQL can be quickly found. The model can understand the user's query intention based on semantic information and support dynamic updating of the vector database to adapt to newly added DDL and SQL content.

[0077] S103: construct model prompt words according to the query question corresponding to the query operation, the associated DDL and the associated SQL, input the model prompt words into a preset large language model, and convert the query question into a corresponding target SQL.

[0078] In the specific implementation, the user query question is combined with the associated DDL and SQL to construct a prompt word that conforms to the input format of a large language model (such as GPT or similar models). The model's powerful semantic understanding and generation capabilities are used to accurately convert natural language query questions into target SQL statements.

[0079] Specifically, the query question is inserted into a preset basic prompt word template to generate a basic prompt word; the associated DDL is inserted into a preset DDL reference prompt word template to generate a DDL prompt word, and the DDL prompt word indicates that the associated DDL is used as a reference for the answer corresponding to the query question; the associated SQL is inserted into a preset SQL reference prompt word template to generate an SQL prompt word, and the SQL prompt word indicates that the associated SQL is used as a reference for the answer corresponding to the query question; the basic prompt word, the DDL prompt word and the SQL prompt word are merged to generate a model prompt word.

[0080] Here, in order to improve the generation accuracy and context understanding ability of the model, the model prompt needs to include the following core information: the user's query question, as the core input target; the database structure related to the user's question (associated DDL), providing context constraints; historical matching SQL (associated SQL), as a reference for generating target SQL.

[0081] Exemplary, basic prompt word template: You try to answer the question raised by the user {user question}, but only provide SQL code; DDL reference prompt word template: Use the following DDL statement as a reference for the answer {DDL retrieval result}; SQL reference prompt word template: Use the following SQL statement as a reference for the answer {SQL retrieval result}.

[0082] Furthermore, the basic prompt words, DDL prompt words and SQL reference prompt words are combined to construct the final model prompt words and input them into the large language model, so that the model can generate the target SQL according to the provided context. The model recognizes the user's query intention through the basic prompt words, understands the relevant table structure and fields according to the DDL prompt words, and uses the associated SQL prompt words to understand the SQL syntax and context conventions. Finally, the target SQL statement that meets the query requirements is comprehensively generated according to the prompt word content.

[0083] S104: Determine the query result of the target SQL in the target business database, and feed back the query result to the user.

[0084] In the specific implementation, starting from the generated target SQL, the final query result is obtained, and the result is desensitized to ensure data security. The query attribute information included in the target SQL is parsed; the query operation is performed in the target business database according to the query attribute information to determine the query result; the query result is desensitized according to the preset sensitive data rules.

[0085] Here, after the target SQL is generated by the large language model, it is necessary to parse it to extract query attribute information such as target table information, query fields, and screening conditions, identify the database tables involved in the SQL (such as the table name that appears after the FROM keyword); extract the fields to be queried in the SQL (such as the field name after the SELECT keyword); and parse the condition information defined in the WHERE clause.

[0086] Furthermore, the legitimacy of the target SQL is verified and the SQL is executed in the target business database. A secure connection is established with the target database through a database driver (such as JDBC, ODBC, etc.), and a query interface (such as executeQuery) is called to execute the target SQL and obtain a query result set.

[0087] It should be noted that it is necessary to define desensitization methods for sensitive fields through preset sensitive data rules to ensure that the data is protected before display. Desensitization methods can include character replacement, field hiding, and range fuzzification. Character replacement masks some characters; field hiding directly removes sensitive fields; range fuzzification ranges precise data values.

[0088] Here, the fields involved in the query results are checked one by one to match the preset sensitive field rules. For example, the query result field user_name may need to have some characters replaced. The result set is processed field by field, line by line, sensitive information is desensitized, and the desensitized data is encapsulated in a standardized format (such as JSON) for easy reading by users.

[0089] In this way, desensitization effectively prevents the leakage of sensitive information, protects user privacy and data compliance, and at the same time, the query results are strictly based on the target SQL and business database to avoid semantic deviation.

[0090] The database query training method based on RAG and large model provided by the embodiment of the present disclosure obtains DDL annotation data and SQL annotation data and standardizes them into training documents, vectorizes the training documents into retrieval knowledge vectors and stores them in a preset vector database; in response to the user's query operation, vectorizes the query operation into a query vector, and retrieves matching associated DDL and associated SQL in the preset vector database according to the query vector; constructs model prompt words according to the query question corresponding to the query operation, the associated DDL and the associated SQL, inputs the model prompt words into the preset large language model, converts the query question into the corresponding target SQL; determines the query result of the target SQL in the target business database, and feeds back the query result to the user. Intelligent conversion from natural language to SQL can be realized, and a vector retrieval enhancement generation process is introduced, which significantly reduces the hallucination problem of the large model and improves the accuracy and security of the generated SQL.

[0091] See also Figure 2 As shown, it is a flowchart of another database query training method based on RAG and large model provided by an embodiment of the present disclosure, the method includes steps S201 to S205, wherein:

[0092] S201, obtaining DDL annotation data and SQL annotation data and standardizing them into training documents, and vectorizing the training documents into retrieval knowledge vectors and storing them in a preset vector database.

[0093] S202: In response to a query operation of a user, vectorize the query operation into a query vector, and retrieve matching associated DDL and associated SQL in the preset vector database according to the query vector.

[0094] S203: construct model prompt words according to the query question corresponding to the query operation, the associated DDL and the associated SQL, input the model prompt words into a preset large language model, and convert the query question into a corresponding target SQL.

[0095] S204, determining the target table information included in the query attribute information; determining whether the user has the authority to query the target table information based on the target table information and the query authority information corresponding to the user; if the user does not have the authority to query the target table information, returning authority exception information to the user.

[0096] S205: Determine the query result of the target SQL in the target business database, and feed back the query result to the user.

[0097] It should be noted that steps S201 to S203 and step S205 are substantially the same as steps S101 to S104, have the same implementation methods and can achieve the same technical effects, and will not be described in detail herein.

[0098] In the specific implementation, after the target SQL is generated, the target table information is determined by parsing the query attribute information, and the query is verified in combination with the user permissions to ensure the security and compliance of the database query. The target table information related to the query operation is parsed from the generated target SQL. In this process, the directly involved table names can be extracted by analyzing the FROM clause, or the tables referenced in the nested query can be identified, or all related table names can be extracted in the multi-table connection query.

[0099] Here, a user permission management module needs to be maintained to record each user's access rights to the database table. Permission information includes user identification (such as user name, user ID); table access rights (such as read-only permission, read-write permission, and specific field permission, etc.). Verify each target table against the user's permission record one by one. If the user does not have access rights to a target table, a permission exception message is generated and returned to the user.

[0100] The content of the permission exception information may include: the restricted table name and the exception reason. The restricted table name is used to specifically indicate which table has insufficient permission; the exception reason may include "no access permission" or "insufficient permission".

[0101] In this way, permissions are checked instantly when users initiate queries, and quick permission exception prompts are provided, which can avoid unauthorized access to data tables and prevent data leakage and illegal operations.

[0102] The database query training method based on RAG and large model provided by the embodiment of the present disclosure obtains DDL annotation data and SQL annotation data and standardizes them into training documents, vectorizes the training documents into retrieval knowledge vectors and stores them in a preset vector database; in response to the user's query operation, vectorizes the query operation into a query vector, and retrieves the matching associated DDL and associated SQL in the preset vector database according to the query vector; constructs a model prompt word according to the query question corresponding to the query operation, the associated DDL and the associated SQL, inputs the model prompt word into the preset large language model, and converts the query question into the corresponding target SQL; determines the target table information included in the query attribute information; determines whether the user has the authority to query the target table information according to the query permission information corresponding to the target table information and the user; if the user does not have the authority to query the target table information, returns permission exception information to the user. Determine the query result of the target SQL in the target business database, and feed back the query result to the user. The intelligent conversion from natural language to SQL can be realized, and the vector retrieval enhancement generation process is introduced, which significantly reduces the hallucination problem of the large model and improves the accuracy and security of the generated SQL.

[0103] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0104] Based on the same inventive concept, the embodiments of the present disclosure also provide a database query training device based on RAG and a large model corresponding to the database query training method based on RAG and a large model. Since the principle of solving the problem by the device in the embodiments of the present disclosure is similar to the above-mentioned database query training method based on RAG and a large model in the embodiments of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0105] See also Figure 3 , Figure 3 FIG. 1 is a schematic diagram of a database query training device based on RAG and a large model provided by an embodiment of the present disclosure. Figure 3 As shown in , the database query training device 300 based on RAG and large model provided by the embodiment of the present disclosure includes:

[0106] The training module 310 is used to obtain DDL annotation data and SQL annotation data and standardize them into training documents, and vectorize the training documents into retrieval knowledge vectors and store them in a preset vector database.

[0107] The associated query module 320 is used to vectorize the query operation into a query vector in response to the user's query operation, and retrieve the matching associated DDL and associated SQL in the preset vector database according to the query vector.

[0108] The prompt word assembly module 330 is used to construct model prompt words according to the query question corresponding to the query operation, the associated DDL and the associated SQL, and input the model prompt words into a preset large language model to generate target SQL.

[0109] The result query module 340 is used to determine the query result of the target SQL in the target business database and feed back the query result to the user.

[0110] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0111] The database query training device based on RAG and large model provided by the embodiment of the present disclosure obtains DDL annotation data and SQL annotation data and standardizes them into training documents, vectorizes the training documents into retrieval knowledge vectors and stores them in a preset vector database; in response to the user's query operation, vectorizes the query operation into a query vector, and retrieves matching associated DDL and associated SQL in the preset vector database according to the query vector; constructs model prompt words according to the query question corresponding to the query operation, the associated DDL and the associated SQL, inputs the model prompt words into the preset large language model, converts the query question into the corresponding target SQL; determines the query result of the target SQL in the target business database, and feeds back the query result to the user. Intelligent conversion from natural language to SQL can be realized, and a vector retrieval enhancement generation process is introduced, which significantly reduces the hallucination problem of the large model and improves the accuracy and security of the generated SQL.

[0112] Corresponds to Figure 1 and Figure 2 The database query training method based on RAG and large model in the present disclosure also provides an electronic device 400, such as Figure 4 FIG. 4 is a schematic diagram of the structure of an electronic device 400 provided in an embodiment of the present disclosure, including:

[0113] Processor 41, memory 42, and bus 43; memory 42 is used to store execution instructions, including memory 421 and external memory 422; memory 421 here is also called internal memory, which is used to temporarily store operation data in processor 41 and data exchanged with external memory 422 such as hard disk. Processor 41 exchanges data with external memory 422 through memory 421. When the electronic device 400 is running, the processor 41 communicates with the memory 42 through bus 43, so that the processor 41 executes Figure 1 and Figure 2 The steps of the database query training method based on RAG and large model.

[0114] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the database query training method based on RAG and large model described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0115] The embodiments of the present disclosure also provide a computer program product, which includes computer instructions. When the computer instructions are executed by a processor, the steps of the database query training method based on RAG and large models described in the above method embodiment can be executed. For details, please refer to the above method embodiment, which will not be repeated here.

[0116] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0117] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0118] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0119] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0120] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0121] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A database query training method based on RAG and large models, characterized in that: include: Obtain DDL annotation data and SQL annotation data and standardize them into training documents, and vectorize the training documents into retrieval knowledge vectors and store them in a preset vector database; In response to a user's query operation, vectorize the query operation into a query vector, and retrieve matching associated DDL and associated SQL in the preset vector database according to the query vector; Construct a model prompt word according to the query question corresponding to the query operation, the associated DDL and the associated SQL, input the model prompt word into a preset large language model, and convert the query question into a corresponding target SQL; Determine the query result of the target SQL in the target business database, and feed back the query result to the user.

2. The method according to claim 1, characterized in that Receive the DDL annotation data and SQL annotation data entered by the user and standardize them into training documents, including: Receiving the DDL annotation data and the SQL annotation data input by the user through a preset online interactive interface; Performing syntax checks on the DDL annotation data and the SQL annotation data respectively, wherein verifying whether the DDL annotation data contains complete annotation information of tables and fields, and verifying the level of detail of the description information corresponding to the SQL annotation data; The DDL annotation data and the SQL annotation data are respectively standardized into corresponding target document formats to generate the training document.

3. The method according to claim 1, characterized in that Vectorizing the training document into a retrieval knowledge vector and storing it in a preset vector database specifically includes: Processing the training document through a preset vectorization algorithm model, and vectorizing the DDL annotation data and the SQL annotation data into retrieval knowledge vectors respectively; The training document is stored in the corresponding table in the preset vector database according to the DDL data and the SQL data respectively.

4. The method according to claim 1, characterized in that: In response to a user's query operation, vectorizing the query operation into a query vector, and searching the preset vector database for matching associated DDL and associated SQL according to the query vector, specifically including: Determine the query question corresponding to the query operation; Vectorizing the query question into the query vector, and searching the DDL table and the SQL table included in the preset vector database according to the query vector; A plurality of the associated DDLs and the associated SQLs whose similarity with the query vector is greater than a preset similarity threshold are determined.

5. The method according to claim 1, characterized in that Constructing a model prompt word according to the query question corresponding to the query operation, the associated DDL, and the associated SQL specifically includes: Inserting the query question into a preset basic prompt word template to generate a basic prompt word; Inserting the associated DDL into a preset DDL reference prompt word template to generate a DDL prompt word, wherein the DDL prompt word indicates that the associated DDL is used as a reference for an answer corresponding to the query question; Inserting the associated SQL into a preset SQL reference prompt word template to generate an SQL prompt word, wherein the SQL prompt word indicates that the associated SQL is used as a reference for an answer corresponding to the query question; The basic prompt word, the DDL prompt word, and the SQL prompt word are combined to generate the model prompt word.

6. The method according to claim 1, characterized in that Determining the query result of the target SQL in the target business database specifically includes: Parsing the query attribute information included in the target SQL; Performing a query operation in the target business database according to the query attribute information to determine the query result; The query results are desensitized according to preset sensitive data rules.

7. The method according to claim 6, characterized in that After inputting the model prompt word into a preset large language model and converting the query question into a corresponding target SQL, the method further includes: Determining target table information included in the query attribute information; Determining whether the user has the authority to query the target table information according to the target table information and the query authority information corresponding to the user; If the user does not have the authority to query the target table information, authority exception information is returned to the user.

8. A database query training device based on RAG and large models, characterized in that: include: A training module is used to obtain DDL annotation data and SQL annotation data and standardize them into training documents, and vectorize the training documents into retrieval knowledge vectors and store them in a preset vector database; An associated query module, used to vectorize the query operation into a query vector in response to a user's query operation, and retrieve matching associated DDL and associated SQL in the preset vector database according to the query vector; A prompt word assembly module, used to construct a model prompt word according to the query question corresponding to the query operation, the associated DDL and the associated SQL, and input the model prompt word into a preset large language model to generate a target SQL; The result query module is used to determine the query result of the target SQL in the target business database and feed back the query result to the user.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the database query training method based on RAG and large model as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the database query training method based on RAG and a large model as described in any one of claims 1 to 7.

Images