Metadata enhancement-based large model Text2SQL (Structured Query Language) implementation method and system

Abstract

The invention discloses a large model Text2SQL (Structured Query Language) implementation method and system based on metadata enhancement, and the method comprises the steps: firstly, constructing a metadata lake for uniformly storing metadata and a data consanguinity map, and the metadata comprises a pre-generated business label; secondly, receiving a natural language query request of a user; next, metadata vector retrieval is performed on the request in a metadata lake to determine a first set of metadata that matches the fuzzy business terms; querying a second group of metadata associated with the first group of metadata, namely an inter-table association relationship, through a data consanguinity map, and combining the two groups of metadata into authoritative metadata; and finally, combining a natural language query request and the authoritative metadata into a context cue word, and inputting the context cue word into the large model to generate an SQL statement. By constructing the complete data context containing the basic metadata and the data consanguinity, the accuracy and credibility of SQL generation are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data query, and mainly relates to a large model Text2SQL implementation method and system based on metadata enhancement. BACKGROUND

[0002] In the data governance process, the development and use of SQL scripts have significant bottlenecks, which are traditionally highly dependent on professional researchers, which not only raises the technical threshold of data governance, but also limits the data processing capabilities of non-professionals.

[0003] With the development of large model technology, Text2SQL technology converts natural language into SQL scripts, which brings new possibilities to reduce this threshold and assists in the modification and rewriting of SQL. However, current Text2SQL technology still faces many serious challenges.

[0004] Firstly, the prominent large model "illusion" phenomenon, that is, when generating SQL, the large model may fabricate non-existent table names, field names or build incorrect inter-table relationships, resulting in invalid SQL. Secondly, when the user's expression is ambiguous, complex or involves professional knowledge, the large model has difficulty accurately understanding the user's real needs. At the same time, when dealing with cross-source and cross-regional data, the metadata format differences of different data sources cause the large model to be unable to accurately obtain and understand metadata information. In order to solve these problems, there have been explorations in the industry, such as using RAG (Retrieval Augmentation Generation) to provide more context for the large model.

[0005] However, these solutions have obvious limitations: traditional RAG solutions often only return single-table metadata when facing complex business scenarios with multiple levels of data association, and cannot obtain associated tables, resulting in the generated SQL missing key association conditions. More importantly, existing explorations do not consider data lineage as a core component of metadata, or only use it for data traceability, and fail to deeply integrate with the "prompt word generation" and "SQL verification" stages of Text2SQL. This results in existing technologies being unable to provide the large model with a complete data context of "basic metadata + lineage logic", so the effect of solving the large model "illusion" problem and improving the credibility of SQL generation is still not ideal. SUMMARY

[0006] In order to solve the technical problems mentioned in the background art, the present application provides a large model Text2SQL implementation method based on metadata enhancement, which comprises:

[0007] Constructing a metadata lake that uniformly stores metadata and data lineage graphs, the metadata at least including data table information and field information; wherein the field information includes data type, field description and pre-generated business tags;

[0008] receiving a natural language query request of a user;

[0009] performing vector retrieval of metadata on the natural language query request in the metadata lake to determine a first set of metadata matching the fuzzy business term vector in the natural language query request, and querying a second set of metadata associated with the first set of metadata through a data bloodline graph to combine the first set of metadata and the second set of metadata into authoritative metadata;

[0010] combining the natural language query request and the authoritative metadata into context prompt words;

[0011] inputting the context prompt words into the large model to generate a SQL statement.

[0012] Further, the metadata is based on the metadata lake engine to collect data table metadata of different data sources, and the collected data table metadata is stored after standardization processing. It solves the problem of insufficient understanding of large models for cross-source data environment, eliminates the differences in metadata formats of different data sources, realizes centralized management and real-time update of metadata, so that the large model can better understand and process data from different data sources, and improve the adaptability in complex data environment.

[0013] Further, the different data sources include a relational database, a NoSQL database, a message queue or a data lake.

[0014] Further, the data bloodline graph specifically includes inter-table association relationships between a plurality of data tables and paths of data flow. It makes the large model understand multi-level data dependency and processing logic, and improves the accuracy of SQL generation in complex scenarios.

[0015] Further, the acquisition of the pre-generated business label specifically includes:

[0016] The metadata is pre-input into a preset metadata feature recognition intelligent agent, the metadata feature recognition intelligent agent combines a preset label library and a preset feature library, and calls a large model for feature recognition and associated acquisition of business labels. It improves the completeness of metadata information, enhances the integrity of metadata information, and provides a high-quality metadata knowledge base for subsequent large models.

[0017] Further, the method further includes:

[0018] Based on the authoritative metadata and the data bloodline graph, the metadata involved in the SQL statement and the inter-table association relationships between the plurality of data tables are matched and verified;

[0019] If the SQL statement fails the check, metadata and feedback information are provided to the large model, and the large model is used to correct the SQL statement. This effectively reduces the generation of invalid SQL caused by large model hallucinations, improves the credibility of SQL generation, can timely discover and correct problems, significantly improves the efficiency of SQL development, improves the matching degree and accuracy of SQL and actual data environment, and reduces data processing failure caused by SQL errors.

[0020] Further, the method further comprises:

[0021] After the SQL statement passes the check, the SQL statement is executed to obtain a data query result.

[0022] Further, the method further comprises:

[0023] Based on the metadata and the large model, the existing SQL statement or the SQL statement that passes the check is modified and / or corrected. This further improves the efficiency and flexibility of SQL processing, provides a more convenient tool and means for data governance work, enhances the SQL processing capability, and further improves the efficiency and quality of SQL processing.

[0024] In another aspect, the present application also provides a large model Text2SQL implementation system based on metadata enhancement, which comprises:

[0025] A metadata lake configured to uniformly store metadata and data bloodline graph, the metadata at least including data table information and field information; wherein the field information includes data type, field description and pre-generated business label;

[0026] A receiving module configured to receive a natural language query request of a user;

[0027] An SQL development platform configured to perform vector retrieval of the natural language query request in the metadata lake, determine a first group of metadata matching the fuzzy business term vector in the natural language query request, and query a second group of metadata associated with the first group of metadata through the data bloodline graph, and combine the first group of metadata and the second group of metadata into authoritative metadata;

[0028] A prompt word construction module configured to combine the natural language query request and the authoritative metadata into contextual prompt words;

[0029] An SQL intelligent agent configured to input the contextual prompt words into the large model to generate a SQL statement.

[0030] Further, the system further comprises:

[0031] The SQL verification module is configured to perform matching verification on metadata involved in the SQL statement and inter-table association relationships among the plurality of data tables based on authoritative metadata and a data bloodline map.

[0032] The SQL correction module is configured to provide the metadata and feedback information to the large model to correct the SQL statement by using the large model when the SQL statement fails the verification.

[0033] The technical effect of the present application is that:

[0034] Lower the SQL development threshold: by converting natural language into SQL, non-professional business personnel can easily perform data query and analysis without mastering complex SQL syntax, greatly reducing the professional threshold of SQL development and expanding the participating population of data development.

[0035] Improve development efficiency: with the assistance of the large model and the metadata, the time and effort cost of manually writing and modifying SQL is reduced. The large model can quickly generate a preliminary SQL, and the metadata verification and auxiliary processing function can timely discover and correct problems, significantly improving the efficiency of SQL development.

[0036] Improve SQL generation quality: effectively reduce the invalid SQL generation caused by the large model "illusion", improve the matching degree and accuracy of SQL and the actual data environment, and reduce the data processing failure caused by SQL errors.

[0037] Improve the accuracy of SQL in complex scenarios: data bloodline as a supplement to metadata enables the large model to understand multi-level data dependencies and processing logic, improving the accuracy of SQL generation in complex business scenarios (more than 3 tables associated, including custom calculation logic) and reducing the rate of invalid SQL.

[0038] Enhance cross-source data processing capability: the unified metadata lake engine realizes the unified management of cross-source and cross-regional data, enabling the large model to better understand and process data from different data sources, and improving the adaptability of the system in complex data environments. BRIEF DESCRIPTION OF DRAWINGS

[0039] The accompanying drawings are included to provide a further understanding of embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the application. Other embodiments and many of the intended advantages of the present application will be readily appreciated as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings. The elements of the drawings are not necessarily to scale relative to each other. Like reference numerals designate corresponding similar parts.

[0040] Figure 1A flowchart of a metadata enhancement-based large model Text2SQL implementation method according to an embodiment of the present application is shown in FIG. 1.

[0041] Figure 2 A structural diagram of a metadata enhancement-based large model Text2SQL implementation system according to an embodiment of the present application is shown in FIG. 2.

[0042] Figure 3 A framework diagram of a metadata enhancement-based large model Text2SQL implementation system according to an embodiment of the present application is shown in FIG. 3.

[0043] Figure 4 A structural diagram of a computer device of an electronic device suitable for implementing an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0044] The present application will be further described below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely intended to explain the related application, and not to limit the application. In addition, it should be noted that only parts related to the application are shown in the drawings for ease of description.

[0045] Figure 1 A flowchart of a metadata enhancement-based large model Text2SQL implementation method according to an embodiment of the present application is shown in FIG. 1. Figure 1 The present application provides a metadata enhancement-based large model Text2SQL implementation method, which comprises the following steps:

[0046] S1, a metadata lake for storing unified metadata and data lineage graphs is constructed, and the metadata at least includes data table information and field information; wherein the field information includes data type, field description and pre-generated business tags;

[0047] Further, the metadata is collected by the metadata lake engine based on the data table metadata of different data sources, and the collected data table metadata is stored after standardization processing. The problem of insufficient understanding of large models for cross-source data environment is solved, the differences in metadata formats of different data sources are eliminated, centralized management and real-time updating of metadata are realized, so that the large model can better understand and process data from different data sources, and the adaptability in complex data environment is improved.

[0048] Further, the metadata lake engine provides efficient metadata query interface and vectorized retrieval interface, which facilitates the large model and other modules to obtain the required metadata information, and provides metadata management interface to support the creation, change and deletion actions of physical data tables through the metadata lake engine, to realize active metadata management and enhance the real-time performance of metadata.

[0049] Further, different data sources include relational databases, NoSQL databases, message queues, or data lakes.

[0050] Further, the data bloodline map specifically includes inter-table association relationships between a plurality of data tables (i.e., static bloodlines) and data flow paths (i.e., dynamic bloodlines). Through the above technical solutions, the multi-level data dependency and processing logic are understood by the large model, and the accuracy of SQL generation in complex scenarios is improved.

[0051] Further, the acquisition of the pre-generated business label specifically includes: pre-inputting the metadata into a preset metadata feature recognition intelligent agent, the metadata feature recognition intelligent agent combining a preset label library and a preset feature library, and calling a large model for feature recognition and associated acquisition of business labels. The metadata information integrity is improved, the metadata information integrity is enhanced, and a high-quality metadata knowledge base is provided for subsequent large models.

[0052] S2, receiving a natural language query request of a user;

[0053] S3, performing vector retrieval of the natural language query request in the metadata lake to determine a first group of metadata matching the fuzzy business term vector in the natural language query request; and querying a second group of metadata associated with the first group of metadata through the data bloodline map, combining the first group of metadata and the second group of metadata as authoritative metadata;

[0054] S4, combining the natural language query request and the authoritative metadata as context prompt words; the prompt words contain detailed data environment description and user demand.

[0055] S5, inputting the context prompt words into the large model to generate a SQL statement. Inputting the above prompt words into the large model can guide the large model to generate a SQL statement that meets the user's demand and is consistent with the data environment.

[0056] S6, based on the authoritative metadata and the data bloodline map, matching and checking the metadata involved in the SQL statement and the inter-table association relationships between a plurality of data tables;

[0057] Further, the matching and checking includes checking whether the table name and the field name involved in the SQL statement exist, whether the data types are matched, whether the multi-table association conditions are reasonable, etc.

[0058] Further, if the SQL statement fails the check, metadata and feedback information are provided to the large model, and the SQL statement is corrected using the large model. Through the above technical solution, the invalid SQL generation caused by the large model illusion is effectively reduced, the credibility of SQL generation is improved, problems can be found and corrected in a timely manner, the efficiency of SQL development is significantly improved, the matching degree and accuracy of SQL and the actual data environment are improved, and the data processing failure caused by SQL errors is reduced.

[0059] S7, after the SQL statement passes the check, the SQL statement is executed to obtain a data query result;

[0060] S8, based on the metadata and the large model, the existing SQL statement or the SQL statement that passes the check is modified, described and / or corrected.

[0061] Further, the modification includes modifying the SQL statement according to the user's requirements, such as adjusting the query condition, changing the aggregation method, etc.; the description includes function description of the SQL statement, explaining the query target, the tables and fields involved, etc.; the correction includes correcting the syntax error and logical error in the SQL statement, and giving modification suggestions.

[0062] Through the above technical solution, the efficiency and flexibility of SQL processing are further improved, a more convenient tool and means are provided for data governance work, the SQL processing capability is enhanced, and the efficiency and quality of SQL processing are further improved.

[0063] The application also provides a large model Text2SQL implementation system based on metadata enhancement. Referring to Figure 2 and Figure 3 The system can be used to execute any of the above method embodiments, and the system comprises:

[0064] A receiving module 100 (corresponding to Figure 3 “Chat” or “SQL editor” in ) configured to receive a user's natural language query request or SQL fragment;

[0065] A metadata lake 200 (corresponding to Figure 3 “Unified metadata storage” in ) configured to store metadata and data bloodline atlas uniformly, and the metadata at least includes data table information and field information; wherein the field information includes data type, field description and pre-generated business label;

[0066] The SQL development platform 300 is configured to perform vector retrieval of metadata in the metadata lake for a natural language query request, determine a first set of metadata matching the vector of fuzzy business terms in the natural language query request, and query a second set of metadata associated with the first set of metadata through a data bloodline graph, and combine the first set of metadata and the second set of metadata into authoritative metadata.

[0067] The prompt word construction module 400 (which can be integrated into the SQL development platform) is configured to combine the natural language query request and the authoritative metadata into contextual prompt words.

[0068] The SQL agent 500 is configured to input the contextual prompt words into a large model to generate an SQL statement.

[0069] Further, the system further comprises:

[0070] The SQL verification module (which can be integrated into the SQL development platform and performs the "SQL verification" function) is configured to perform matching verification on the metadata involved in the SQL statement and the inter-table association relationship between the plurality of data tables based on the authoritative metadata and the data bloodline graph.

[0071] The SQL correction module (which can be integrated into the SQL development platform) is configured to provide metadata and feedback information to the large model to correct the SQL statement using the large model when the SQL statement fails the verification.

[0072] Further, the system can further comprise a metadata collector and a metadata feature recognition agent. The metadata collector is used to collect metadata from different data sources (relational, message queue, data lake, NoSQL database). The metadata feature recognition agent is used to pre-generate business labels.

[0073] As an example, in the present embodiment, taking the user inputting a natural language query "query all A-class customers in XX region in the last quarter Order total amount" in the "Chat" interface (belonging to the SQL application) as an example, the specific implementation steps are as follows:

[0074] Step one, receiving and intent recognition (S2), the user inputs a natural language query request in the "Chat" interface. The "Chat" application sends the "natural language" request to the "SQL development platform".

[0075] Step two, semantic retrieval (first stage of S3: query the first set of metadata), the "SQL development platform" receives the request and initiates a query to the "metadata vector retrieval" module to understand the fuzzy business terms such as "A-class customers", "order total amount", "XX region", etc. The "vector retrieval" returns the first set of metadata that is most similar in semantics, for example:

[0076] "Class A Customers" -> associated with the customer_level (customer level) field of the customers (customer table).

[0077] "Total Order Amount" is related to the order_amount field of the orders table.

[0078] "XX region" is associated with the city field of the customers table.

[0079] Step 3, Logical Relationship (Second Stage of S3: Querying the Second Set of Metadata): Based on the first set of metadata (involving the customers and orders tables), the platform initiates a query to the "Data Lineage Graph Retrieval" module, providing GraphRAG capabilities. "Lineage Retrieval" returns the inter-table relationship between these two tables, i.e., the second set of metadata, for example: customers.customer_id related to orders.customer_id.

[0080] Step 4: Precise Information Acquisition (S3's Third Stage: Combining Authoritative Metadata). The platform initiates a precise query to the "Metadata Query" module to extract authoritative metadata for the fields involved in the first and second sets of metadata. Authoritative metadata includes pre-stored data types, business tags, and field descriptions, for example:

[0081] Table 1 (customers): Fields: customer_id (type: int, label: primary key), customer_level (type: string, label: customer level, description: A / B / C), city (type: string, label: city)

[0082] Table 2 (orders): Fields: order_id (type: int, label: primary key), customer_id (type: int, label: foreign key), order_amount (type: decimal, label: core KPI), order_date (type: timestamp, label: business date). The SQL Development Platform combines this information into the final authoritative metadata.

[0083] Step 5: Constructing Prompt Words and Generating SQL (S4, S5) The "SQL Development Platform" packages all information (natural language in S2, authoritative metadata in S3) and sends it to the "SQL Agent." The "SQL Agent" constructs high-quality contextual prompt words and sends them to the "Base Model," which receives the prompt words and generates the initial SQL statement.

[0084] Step six, SQL verification and correction (S6) The "SQL development platform" receives the initial SQL and starts the "SQL verification" process. The platform calls the metadata lake (for example, the "metadata query" module) to perform 100% accurate matching verification on the table names (customers), field names (city, customer_level) and inter-table association relationships (customers.customer_id = orders.customer_id) in the initial SQL. If the large model "hallucinates" and writes customer_rank (while the authoritative metadata is customer_level), the verification module will find a mismatch, and the system will provide the authoritative metadata and feedback information, requiring the large model to correct it.

[0085] Step seven, execute and output results (S7) After the SQL verification passes, the "SQL development platform" sends the SQL for execution (for example, through the "SQL editor" "SQL verification & execution" path) and obtains the data query results. The platform packages the results into "output agent results" and displays them to the user through the "Chat" interface.

[0086] The present application can achieve the following technical effects:

[0087] Lower the threshold of SQL development: By converting natural language into SQL, non-professional business personnel can easily perform data query and analysis without mastering complex SQL syntax, greatly reducing the professional threshold of SQL development and expanding the participating population of data development.

[0088] Improve development efficiency: With the assistance of large models and metadata, the time and effort cost of manually writing and modifying SQL is reduced. Large models can quickly generate preliminary SQL, and metadata verification and auxiliary processing functions can timely discover and correct problems, significantly improving the efficiency of SQL development.

[0089] Improve SQL generation quality: effectively reduce the invalid SQL generation caused by large model "hallucination", improve the matching degree and accuracy of SQL and actual data environment, and reduce the data processing failure caused by SQL errors.

[0090] Improve the accuracy of complex scene SQL: Data lineage as a supplement to metadata enables large models to understand multi-level data dependencies and processing logic, improving the accuracy of SQL generation in complex business scenarios (such as more than 3 table associations, including custom calculation logic), and reducing the rate of invalid SQL.

[0091] Enhanced cross-source data processing capability: The unified metadata lake engine realizes unified management of cross-source and cross-regional data, enabling large models to better understand and process data from different data sources, and improving the adaptability of the system in complex data environments.

[0092] Reference will now be made to the following description Figure 4 which shows a structural schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the present application. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functions and use range of embodiments of the present application.

[0093] As Figure 4 shown, the computer system includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 402 or programs loaded from a storage portion 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the system 400 are also stored. The CPU 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0094] The following components are connected to the I / O interface 405: an input portion 406 including a keyboard, a mouse, and the like; an output portion 407 including a liquid crystal display (LCD), a speaker, and the like; a storage portion 408 including a hard disk, and the like; and a communication portion 409 including a network interface card such as a LAN card, a modem, and the like. The communication portion 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as necessary. A removable recording medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is attached to the drive 410 as necessary, so that a computer program read therefrom is installed in the storage portion 408 as necessary.

[0095] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program in accordance with embodiments of the present application. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable storage medium, the computer program comprising program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communications section 409 and / or installed from the removable media 41 1. When the computer program is executed by the central processing unit (CPU) 401, the above described functions defined in the methods of the present application are performed. It is noted that the computer readable storage medium of the present application can be either a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium can be, for example and without limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, the computer readable storage medium can be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present application, the computer readable signal medium can include a computer readable program code carried by a data signal in a baseband or as part of a carrier wave. Such a propagated signal can take any of a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. Computer readable signal medium can be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0096] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0097] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0098] The modules involved in the embodiments of the present application can be implemented by software or by hardware.

[0099] As another aspect, the application also provides a computer readable storage medium, which can be included in the electronic device described in the above embodiments, or can exist independently without being assembled into the electronic device. The computer readable storage medium carries one or more programs, when the one or more programs are executed by the electronic device, the electronic device is caused to: construct a metadata lake of unified storage metadata and data lineage graph, the metadata at least includes data table information and field information; wherein the field information includes data type, field description and pre-generated business label; receive a natural language query request of a user; perform vector retrieval of the metadata lake on the natural language query request to determine a first group of metadata matched with a fuzzy business term vector in the natural language query request; query a second group of metadata associated with the first group of metadata through the data lineage graph, and combine the first group of metadata and the second group of metadata into authoritative metadata; combine the natural language query request and the authoritative metadata into a context prompt word; input the context prompt word into a large model to generate a SQL statement.

[0100] The specific embodiments of the application are described above, but the protection scope of the application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the application, which should be covered within the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

[0101] In the description of the application, it should be understood that the terms 'upper', 'lower', 'inner', 'outer' and the like indicate the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the application. The word 'comprising' does not exclude the presence of elements or steps not listed in the claims. The word 'a' or 'an' in front of an element does not exclude the presence of multiple such elements. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that the combination of these measures cannot be used to advantage. Any reference signs in the claims should not be interpreted as limiting the scope.

Claims

1. A method for implementing large-scale Text2SQL models based on metadata enhancement, characterized in that, include: Construct a metadata lake that unifies the storage of metadata and data lineage graphs. The metadata includes at least data table information and field information. The field information includes data type, field description, and pre-generated business tags. Receive natural language query requests from users; Vector retrieval of metadata for the natural language query request is performed in the metadata lake to determine the first set of metadata that matches the fuzzy business term vector in the natural language query request; and the second set of metadata associated with the first set of metadata is queried through the data lineage graph, and the first set of metadata and the second set of metadata are combined into authoritative metadata; The natural language query request and the authoritative metadata are combined into contextual prompts; The contextual prompts are input into the large model to generate SQL statements.

2. The method according to claim 1, characterized in that, The metadata is collected from data table metadata from different data sources based on the metadata lake engine, and the collected data table metadata is standardized and then stored.

3. The method according to claim 2, characterized in that, The different data sources include relational databases, NoSQL databases, message queues, or data lakes.

4. The method according to claim 1, characterized in that, The data lineage map specifically includes the inter-table relationships between multiple data tables, as well as the data flow paths.

5. The method according to claim 1, characterized in that, The acquisition of the pre-generated business tags specifically includes: Metadata is pre-input into a preset metadata feature recognition intelligent agent. The metadata feature recognition intelligent agent combines a preset tag library and a preset feature library, and calls the large model to perform feature recognition and obtain the association of the business tags.

6. The method according to claim 1, characterized in that, The method further includes: Based on the authoritative metadata and the data lineage graph, the metadata involved in the SQL statement and the inter-table relationships between multiple data tables are matched and verified. If the SQL statement fails validation, the metadata and feedback information are provided to the large model, which then corrects the SQL statement.

7. The method according to claim 6, characterized in that, The method further includes: After the SQL statement passes the validation, the SQL statement is executed to obtain the data query results.

8. The method according to claim 1, characterized in that, The method further includes: Based on the metadata and the large model, existing SQL statements or SQL statements that have passed validation are modified and / or corrected.

9. A large-scale Text2SQL implementation system based on metadata enhancement, characterized in that, include: The metadata lake is configured to uniformly store metadata and data lineage graphs. The metadata includes at least data table information and field information; wherein, the field information includes data type, field description, and pre-generated business tags. The receiving module is configured to receive users' natural language query requests. The SQL development platform is configured to perform vector retrieval of metadata for the natural language query request in the metadata lake, determine a first set of metadata that matches the fuzzy business term vector in the natural language query request, and query a second set of metadata associated with the first set of metadata through the data lineage graph, and combine the first set of metadata and the second set of metadata into authoritative metadata. The prompt word construction module is configured to combine the natural language query request and the authoritative metadata into context prompt words; The SQL agent is configured to input the contextual prompts into a large model to generate SQL statements.

10. The system according to claim 9, characterized in that, The system also includes: The SQL verification module is configured to match and verify the metadata involved in the SQL statement and the inter-table relationships between multiple data tables based on the authoritative metadata and the data lineage graph. The SQL correction module is configured to provide the metadata and feedback information to the large model when the SQL statement fails validation, so as to use the large model to correct the SQL statement.

Images