Natural language SQL (Structured Query Language) conversion method based on data platform and large language model

Abstract

The invention provides a natural language SQL (Structured Query Language) conversion method based on a data platform and a large language model, which relates to the technical field of data processing and comprises the following steps of: performing vectorization storage on metadata, blood relationship, indexes and historical query information, performing intention judgment and vector matching on a user query request, and determining a query condition based on a matching result. An association path is constructed according to data consanguinity, and an optimal SQL statement is generated by utilizing a historical query optimization table connection sequence and combining index matching. According to the method, intelligent conversion from the natural language to the efficient SQL can be realized, the query efficiency is improved, and the use threshold of a user is reduced.

Description

Technical Field

[0001] The present invention relates to data processing technology, and in particular to a natural language SQL conversion method based on a data platform and a large language model. Background Art

[0002] With the rapid development of big data technology, enterprises have accumulated a large amount of data assets. In order to make full use of these data resources for analytical decision-making, data analysts need to frequently write SQL query statements to obtain the required data. However, for business personnel who are not familiar with SQL syntax, it is very difficult to directly write SQL query statements. Although there are some natural language to SQL tools on the market, these tools can often only handle simple single-table query scenarios and still face many challenges when handling complex multi-table association queries.

[0003] The existing technology mainly has the following deficiencies: Existing natural language-to-SQL tools lack an in-depth understanding of business semantics and are unable to accurately identify the business rules and indicator definitions implicit in user query intent, resulting in a deviation between the generated SQL query results and the actual needs of users.

[0004] When processing multi-table association queries, existing technologies usually adopt fixed table connection strategies, fail to fully utilize historical query experience and database index information to optimize query performance, resulting in low query execution efficiency.

[0005] When processing incomplete or ambiguous query requests, the existing technology lacks an effective interactive mechanism to clarify and supplement the query conditions, and cannot guarantee the accuracy and completeness of the generated SQL query statements, thus affecting the availability of the query results. Summary of the invention

[0006] The embodiment of the present invention provides a natural language SQL conversion method based on a data platform and a large language model, which can solve the problems in the prior art.

[0007] According to a first aspect of the embodiments of the present invention, Provides natural language SQL conversion methods based on data platforms and large language models, including: The obtained metadata information, data lineage information, business indicator information and historical query information are converted into vector form through the data service API and stored in the knowledge vector library; the query request input by the user is judged by natural language processing technology, and the extracted intention information is converted into a query vector; the query vector is matched with the metadata vector, lineage vector, indicator vector and historical query vector in the knowledge vector library for similarity; the data table, field and association relationship to be queried are determined based on the matching results; when the matching results show that the query conditions are incomplete, a supplementary query is generated according to the business indicator information in the knowledge vector library; the user's feedback information on the supplementary query is obtained, and the query vector is updated; the similarity matching process is repeated until the complete query conditions are obtained; Based on the query vector and the matching result, the structural information and field definition of the target data table are extracted from the knowledge vector library; an initial association path is constructed according to the data flow path in the data lineage information; based on the data table in the initial association path, historical query records containing the same data table combination are retrieved from the historical query information; the historical query records are sorted according to the query execution time, and the table connection sequence in the historical query record with the shortest execution time is selected as the optimized table association sequence; According to the index information in the metadata information, index matching is performed on the data screening conditions and table connection conditions involved, and the query method with the highest index coverage is selected as the index matching result; based on the table association order and the index matching result, a complete SQL query statement including the data screening conditions, multi-table connection relationships and target fields is generated; the SQL query statement is executed and the query result is recorded; and the SQL query statement, the query vector, the query execution time, the associated path and the query result are updated to the historical query information.

[0008] The metadata information includes data table structure, field definition, index information, constraints, data types and field comments; The data lineage information includes data flow paths, upstream and downstream associations of data objects, and task dependencies; The business indicator information includes sales statistics, conversion rate calculation and inventory turnover rate; The historical query information includes SQL query statements, query frequency and query execution time.

[0009] Determining the data table, field and association relationship to be queried based on the matching results; when the matching results show that the query condition is incomplete, generating a supplementary query according to the business indicator information in the knowledge vector library; obtaining user feedback information on the supplementary query, and updating the query vector includes: The query vector is matched with the vectors in the knowledge vector library by using a similarity calculation method. The query vector and the vectors in the knowledge vector library are matched in multiple dimensions by cosine similarity calculation, and the matching result is determined according to a set similarity threshold; Based on the matching results, a query graph is constructed, wherein the query graph includes a vertex set representing a data table and an edge set representing an association relationship between tables, and a connectivity index of the query graph is calculated, wherein the connectivity index represents the integrity of the graph through the relationship between the number of edges and the number of vertices; when the connectivity index is less than a preset connectivity threshold, a data table node that needs to supplement the association condition is marked; Verify the validity of the fields involved in the query, and calculate the field validity weight based on the field relevance, query frequency and null value rate. The field relevance represents the degree of association between the field and the business indicator, the query frequency represents the frequency of use of the field in historical queries, and the null value rate represents the data quality of the field; Based on the field validity weight, a hierarchical query strategy is constructed, wherein the hierarchical query strategy includes a core business condition confirmation layer, an association relationship supplement layer, and a query constraint refinement layer, and each layer is assigned a different importance coefficient; the query priority is calculated according to the influence of the business indicator on the supplementary condition, and the corresponding supplementary query is generated according to the query priority; Obtain user feedback information on the supplementary query, calculate vector update strength based on the feedback information, where the vector update strength is determined by the number of feedbacks, the feedback importance, and the learning rate; enhance and normalize the query vector according to the vector update strength to generate an updated query vector.

[0010] Based on the query vector and the matching result, extracting the structure information and field definition of the target data table from the knowledge vector library; constructing the initial association path according to the data flow path in the data lineage information includes: Decomposing the query vector into an entity vector and a relationship vector, and performing a linear combination of the entity vector and the relationship vector based on a preset weight coefficient to generate a target query vector; calculating the relevance score between the target query vector and each data table in the knowledge vector library, wherein the relevance score is obtained by weighted calculation of the entity matching score, the relationship matching score, and the field coverage, and screening the data table whose relevance score is greater than the relevance threshold as a candidate data table; Constructing a structural similarity matrix for the candidate data tables, wherein each element value in the structural similarity matrix is ​​obtained by calculating the ratio of the intersection and the union of the field sets of the corresponding two data tables; calculating a semantic feature vector for each field based on the structural similarity matrix, performing weighted summation on the semantic feature vectors to obtain a field semantic vector, and calculating the field semantic association between the candidate data tables based on the field semantic vectors; A data lineage directed weighted graph is constructed based on the semantic association of the fields, wherein the nodes of the directed weighted graph represent candidate data tables, the edges represent associations between tables, and the edge weights are obtained by weighted calculation of data dependency strength, data flow frequency, and time correlation; and the path cost values ​​of all association paths in the graph are calculated based on the edge weights and the pre-calculated reliability coefficients; Selecting a path with the smallest path cost from the association paths as a candidate association path, calculating a quality score of the candidate association path, wherein the quality score is obtained by the ratio of the cumulative multiplication of all edge weight values ​​in the path to the sum of squares of connection costs; and calculating the redundancy of the candidate association path, wherein the redundancy is determined based on the ratio of a required edge set to maintain the association relationship of the data table to an actual edge set; When the quality score is greater than a preset quality threshold and the redundancy is less than a preset redundancy threshold, the candidate association path is determined as the final association path; otherwise, a path with the second smallest path cost is selected from the remaining candidate association paths as a new candidate association path, and the quality score calculation and redundancy calculation are repeatedly performed until a final association path that meets the quality and redundancy requirements is found.

[0011] According to the index information in the metadata information, index matching is performed on the data screening conditions and table connection conditions involved, and the query method with the highest index coverage is selected as the index matching result; based on the table association order and the index matching result, a complete SQL query statement containing data screening conditions, multi-table connection relationships and target fields is generated, including: Obtain the index information and query condition set of the data table, calculate the matching degree between each index and the query condition, calculate the index usage cost according to the index storage overhead, access time and maintenance cost, and select the index combination with the minimum index usage cost and index coverage exceeding the preset coverage threshold as the candidate index set; Analyze the query conditions based on the candidate index set, calculate the selectivity score of the data screening conditions, and determine the selectivity score according to the ratio of the amount of data that meets the screening conditions to the total amount of data in the table; evaluate the table connection conditions, calculate the connection cost value based on the table size involved in the connection, the connection index cardinality and the connection type, and select the connection condition combination whose connection cost value is lower than the preset cost threshold; Using the connection condition combination to construct a multi-table connection tree, using the connection cost value as the weight of the connection edge, calculating the size of the intermediate result set of different connection sequences, sorting all connection sequences based on the selectivity and calculation cost of the connection steps, and selecting the sequence with the smallest total cost as the optimal connection sequence; Generate an initial SQL query statement according to the optimal connection sequence and the candidate index set, and calculate the query integrity score of the query statement, wherein the integrity score includes three dimensions: query field coverage, connection condition integrity, and index utilization; at the same time, build a performance prediction model based on theoretical execution time and IO overhead to calculate the expected execution efficiency of the query statement; When the query integrity score is lower than a preset score threshold or the expected execution efficiency is lower than a preset efficiency threshold, the adaptive tuning process is started by adjusting the connection sequence, updating the index selection, and optimizing the query structure until the performance requirements are met or the maximum number of tuning times is reached.

[0012] Using the connection condition combination to construct a multi-table connection tree, using the connection cost value as the weight of the connection edge, calculating the size of the intermediate result set of different connection sequences, sorting all possible connection sequences based on the selectivity and calculation cost of the connection steps, and selecting the sequence with the smallest total cost as the optimal connection sequence includes: A connection relevance evaluation model is constructed based on the number of foreign key associations and data relevance between data tables to calculate the connection relevance score between any two data tables. The connection relevance score is determined by the ratio of the number of foreign key associations to the number of table records and the data relevance coefficient. Calculate the weight value of the connection edge according to the connection relevance score, the weight value is obtained by weighted calculation of the connection operation cost, input and output overhead and processor calculation overhead, and use the weight value as the weight attribute of the connection tree edge; Calculate the cumulative execution cost for all connection sequences, the cumulative execution cost includes the connection operation cost, the intermediate result transmission cost and the storage cost, and calculate the sequence execution efficiency score by combining the selection rate and processing cost of the connection steps; Selecting a sequence with the highest execution efficiency score from all connection sequences as a candidate sequence, and calculating an expected resource consumption value of the candidate sequence, wherein the expected resource consumption value includes a processor usage rate, a memory occupancy rate, and a network bandwidth usage rate; When the expected value of resource consumption exceeds the system capacity threshold, the sequence optimization process is triggered to reduce resource consumption by adjusting the connection order, splitting the connection steps or parallel processing until the optimal connection sequence that meets the system resource constraints is found.

[0013] A second aspect of an embodiment of the present invention provides a natural language SQL conversion system based on a data platform and a large language model, including: The first unit is used to convert the acquired metadata information, data lineage information, business indicator information and historical query information into vector form through the data service API and store them in the knowledge vector library; use natural language processing technology to identify the intent of the query request input by the user, and convert the extracted intent information into a query vector; perform similarity matching on the query vector with the metadata vector, lineage vector, indicator vector and historical query vector in the knowledge vector library; determine the data table, field and association relationship to be queried based on the matching results; when the matching results show that the query conditions are incomplete, generate a supplementary query according to the business indicator information in the knowledge vector library; obtain user feedback information on the supplementary query and update the query vector; repeat the similarity matching process until the complete query conditions are obtained; The second unit is used to extract the structural information and field definition of the target data table from the knowledge vector library based on the query vector and the matching result; construct an initial association path according to the data flow path in the data lineage information; retrieve historical query records containing the same data table combination from the historical query information based on the data table in the initial association path; sort the historical query records according to the query execution time, and select the table connection order in the historical query record with the shortest execution time as the optimized table association order; The third unit is used to perform index matching on the data screening conditions and table connection conditions involved according to the index information in the metadata information, and select the query method with the highest index coverage as the index matching result; based on the table association order and the index matching result, generate a complete SQL query statement including the data screening conditions, multi-table connection relationships and target fields; execute the SQL query statement and record the query results; and update the SQL query statement, the query vector, the query execution time, the associated path and the query results to the historical query information.

[0014] A third aspect of the embodiments of the present invention An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0015] A fourth aspect of the embodiments of the present invention is: A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0016] The beneficial effects of this application are as follows: The present invention vectorizes and stores metadata, data lineage, business indicators and historical query information, and uses natural language processing technology to perform intent discrimination and similarity matching, which can accurately understand the user's query intent and quickly locate relevant data tables and fields. When the query conditions are incomplete, the system can also generate supplementary queries based on business indicator information to ensure that complete query conditions are obtained, thereby improving the accuracy and efficiency of the query.

[0017] The present invention uses data lineage information to construct the initial association path and optimizes the table association order by analyzing historical query records, which can effectively reduce the complexity and execution time of data query. At the same time, by matching and optimizing index information and selecting the optimal query method, the query performance is further improved.

[0018] The present invention updates the generated SQL query statements, query vectors, execution time, associated paths and query results to the historical query information, forming a continuously optimized closed-loop system. This method can not only continuously improve query efficiency, but also provide valuable references for subsequent queries, thereby achieving continuous optimization of query performance and continuous improvement of the system intelligence level. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flowchart of a natural language SQL conversion method based on a data platform and a large language model according to an embodiment of the present invention; Figure 2 A logical diagram of a natural language SQL conversion method based on a data platform and a large language model according to an embodiment of the present invention Figure 3 The figure is a schematic diagram of the structure of a natural language SQL conversion system based on a data platform and a large language model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0022] Figure 1 and Figure 2They are respectively a flow chart and a logic chart of a natural language SQL conversion method based on a data platform and a large language model according to an embodiment of the present invention, wherein the method comprises: S101. Convert the acquired metadata information, data lineage information, business indicator information and historical query information into vector form through the data service API and store them in the knowledge vector library; use natural language processing technology to identify the intent of the query request input by the user, and convert the extracted intent information into a query vector; perform similarity matching on the query vector with the metadata vector, lineage vector, indicator vector and historical query vector in the knowledge vector library; determine the data table, field and association relationship to be queried based on the matching results; when the matching results show that the query conditions are incomplete, generate a supplementary query based on the business indicator information in the knowledge vector library; obtain user feedback information on the supplementary query and update the query vector; repeat the similarity matching process until the complete query conditions are obtained; S102. Based on the query vector and the matching result, extract the structural information and field definition of the target data table from the knowledge vector library; construct an initial association path according to the data flow path in the data lineage information; based on the data table in the initial association path, retrieve the historical query records containing the same data table combination from the historical query information; sort the historical query records according to the query execution time, and select the table connection order in the historical query record with the shortest execution time as the optimized table association order; S103. According to the index information in the metadata information, index matching is performed on the data screening conditions and table connection conditions involved, and the query method with the highest index coverage is selected as the index matching result; based on the table association order and the index matching result, a complete SQL query statement including data screening conditions, multi-table connection relationships and target fields is generated; the SQL query statement is executed and the query results are recorded; the SQL query statement, the query vector, the query execution time, the associated path and the query results are updated to the historical query information.

[0023] In an optional implementation, the metadata information includes data table structure, field definition, index information, constraints, data types and field annotations; The data lineage information includes data flow paths, upstream and downstream associations of data objects, and task dependencies; The business indicator information includes sales statistics, conversion rate calculation and inventory turnover rate; The historical query information includes SQL query statements, query frequency and query execution time.

[0024] First, we build a metadata collection module to obtain table structure information of various data sources through database connectors. For relational databases, we use the JDBC driver to read the data dictionary and extract basic information such as table name, field name, field type, default value, whether it is empty, etc. At the same time, we parse the DDL statement to obtain integrity constraint information such as primary key constraints, foreign key constraints, and unique constraints.

[0025] When obtaining data lineage relationships, first parse the ETL job configuration file to extract the input and output table information in the data processing task. By building a directed graph model, the dependencies between tables are stored as edges, and the conversion rules are recorded at the same time. For stored procedures and views, by parsing their definition statements, the source and target tables involved are identified, and the lineage relationship graph is supplemented and improved.

[0026] The collection of business indicator information needs to be combined with specific business scenarios. Taking sales statistics as an example, by configuring the indicator caliber, determine the statistical period, summary dimension, calculation rules and other elements. Create a sales fact table in the data warehouse to record fields such as order amount, payment status, refund amount, etc. Through the pre-aggregation view, calculate sales according to different dimensions (such as date, region, category).

[0027] For conversion rate, it is necessary to record the status changes of each link in the user behavior fact table. For example, in the e-commerce scenario, record key nodes such as browsing products, adding to shopping carts, submitting orders, and successful payment. Through behavior sequence analysis, calculate the conversion ratio between each link. At the same time, combine user grouping to analyze the conversion performance of users with different characteristics.

[0028] The calculation of inventory turnover rate is based on the inventory snapshot table and the inventory in and out flow table. The daily balance quantity and amount are recorded in the inventory snapshot table, and the sales outflow volume during the period is calculated through the inventory in and out flow table. Combined with the purchase inflow record, the replenishment timeliness can be analyzed and the inventory management strategy can be optimized.

[0029] The collection of historical query information is achieved by deploying monitoring components on the database server. Record the original SQL statement, execution timestamp, response time, number of affected rows and other execution details. By establishing a query pattern library, similar SQL statements are classified and the execution frequency of various queries is counted. Index optimization suggestions are established for high-frequency queries to improve query performance.

[0030] This application can achieve: By comprehensively collecting and analyzing metadata information, we can accurately grasp the current status of data assets, provide basic support for data governance, and improve data quality and efficiency. Based on data lineage analysis, we can quickly locate the source of data anomalies, assess the impact of changes, reduce operation and maintenance risks, and improve the reliability and maintainability of data processing. Through standardized indicator systems and historical query analysis, we can effectively support business decisions, optimize system performance, improve user experience, and achieve continuous mining and application of data value.

[0031] In an optional implementation, determining the data table, field, and association relationship to be queried based on the matching result; when the matching result shows that the query condition is incomplete, generating a supplementary query according to the business indicator information in the knowledge vector library; obtaining user feedback information on the supplementary query, and updating the query vector includes: The query vector is matched with the vectors in the knowledge vector library by using a similarity calculation method. The query vector and the vectors in the knowledge vector library are matched in multiple dimensions by cosine similarity calculation, and the matching result is determined according to a set similarity threshold; Based on the matching results, a query graph is constructed, wherein the query graph includes a vertex set representing a data table and an edge set representing an association relationship between tables, and a connectivity index of the query graph is calculated, wherein the connectivity index represents the integrity of the graph through the relationship between the number of edges and the number of vertices; when the connectivity index is less than a preset connectivity threshold, a data table node that needs to supplement the association condition is marked; Verify the validity of the fields involved in the query, and calculate the field validity weight based on the field relevance, query frequency and null value rate. The field relevance represents the degree of association between the field and the business indicator, the query frequency represents the frequency of use of the field in historical queries, and the null value rate represents the data quality of the field; Based on the field validity weight, a hierarchical query strategy is constructed, wherein the hierarchical query strategy includes a core business condition confirmation layer, an association relationship supplement layer, and a query constraint refinement layer, and each layer is assigned a different importance coefficient; the query priority is calculated according to the influence of the business indicator on the supplementary condition, and the corresponding supplementary query is generated according to the query priority; Obtain user feedback information on the supplementary query, calculate vector update strength based on the feedback information, where the vector update strength is determined by the number of feedbacks, the feedback importance, and the learning rate; enhance and normalize the query vector according to the vector update strength to generate an updated query vector.

[0032] In the process of matching query vectors with knowledge vector libraries, the first step is to vectorize the input natural language query. The query text is converted into a high-dimensional vector representation through a pre-trained language model, and the vector dimension is usually 768 or 1024. The knowledge vector library stores vector representations of standard descriptions of business indicators, associated data tables, field definitions, and other information.

[0033] When performing vector matching, block calculation is used to improve efficiency. The knowledge vector library is divided into blocks according to the business domain, and each block contains related business indicator information. The preliminary matching results are obtained by calculating the cosine similarity between the query vector and the vectors in each block. The similarity threshold is set to 0.75, and only matching items with a similarity greater than the threshold are retained.

[0034] When building a query graph based on the matching results, the identified data tables are used as the vertices of the graph, and the associations between the tables are used as edges. For example, if the order table and the user table are associated through the user ID, an edge is added between the corresponding vertices. The connectivity index is obtained by calculating the ratio of the number of edges to the number of vertices. When the index is less than 0.8, it means that the query graph is not complete and the association conditions need to be supplemented.

[0035] For the fields involved in the query, their effectiveness is evaluated from multiple dimensions. The field relevance is calculated by the co-occurrence frequency with the business indicator, the query frequency is based on recent query log statistics, and the null value rate is calculated through sampling data. The weighted average of these three indicators is used as the effectiveness weight of the field.

[0036] The layered query strategy divides the supplementary conditions into three layers: the importance coefficient of the core business condition confirmation layer is 0.5, the association supplement layer is 0.3, and the query constraint refinement layer is 0.2. In each layer, the query priority is determined based on the field validity weight and the degree of business impact. For example, in the order analysis scenario, first confirm the order time range, then supplement the association conditions of the user dimension, and finally refine the constraints such as the order status.

[0037] After obtaining user feedback, calculate the vector update strength. The number of feedbacks indicates the number of times the user confirms the same question. The importance of the feedback is determined based on the user's selection operation. The learning rate is initially set to 0.1 and decreases with the number of feedbacks. Adjust the corresponding dimension of the query vector according to the update strength to ensure that the updated vector still meets the normalization constraint.

[0038] This application can achieve: Through multi-dimensional vector matching and dynamic threshold adjustment, the accuracy of query intent understanding is improved, query errors caused by ambiguous understanding are reduced, and the system can more accurately identify users' real query needs.

[0039] The use of a hierarchical query strategy and priority sorting mechanism reduces the cost of user interaction, avoids redundant confirmation processes, improves the efficiency of query condition supplementation, and ensures the integrity and rationality of the supplementary conditions.

[0040] The dynamic vector update mechanism based on user feedback realizes the continuous optimization of query understanding ability. The system can adaptively adjust the query strategy, improve the accuracy of query results and user satisfaction, and reduce the workload of system maintenance.

[0041] In an optional implementation, based on the query vector and the matching result, extracting the structure information and field definition of the target data table from the knowledge vector library; constructing the initial association path according to the data flow path in the data lineage information includes: Decomposing the query vector into an entity vector and a relationship vector, and performing a linear combination of the entity vector and the relationship vector based on a preset weight coefficient to generate a target query vector; calculating the relevance score between the target query vector and each data table in the knowledge vector library, wherein the relevance score is obtained by weighted calculation of the entity matching score, the relationship matching score, and the field coverage, and screening the data table whose relevance score is greater than the relevance threshold as a candidate data table; Constructing a structural similarity matrix for the candidate data tables, wherein each element value in the structural similarity matrix is ​​obtained by calculating the ratio of the intersection and the union of the field sets of the corresponding two data tables; calculating a semantic feature vector for each field based on the structural similarity matrix, performing weighted summation on the semantic feature vectors to obtain a field semantic vector, and calculating the field semantic association between the candidate data tables based on the field semantic vectors; A data lineage directed weighted graph is constructed based on the semantic association of the fields, wherein the nodes of the directed weighted graph represent candidate data tables, the edges represent associations between tables, and the edge weights are obtained by weighted calculation of data dependency strength, data flow frequency, and time correlation; and the path cost values ​​of all association paths in the graph are calculated based on the edge weights and the pre-calculated reliability coefficients; Selecting a path with the smallest path cost from the association paths as a candidate association path, calculating a quality score of the candidate association path, wherein the quality score is obtained by the ratio of the cumulative multiplication of all edge weight values ​​in the path to the sum of squares of connection costs; and calculating the redundancy of the candidate association path, wherein the redundancy is determined based on the ratio of a required edge set to maintain the association relationship of the data table to an actual edge set; When the quality score is greater than a preset quality threshold and the redundancy is less than a preset redundancy threshold, the candidate association path is determined as the final association path; otherwise, a path with the second smallest path cost is selected from the remaining candidate association paths as a new candidate association path, and the quality score calculation and redundancy calculation are repeatedly performed until a final association path that meets the quality and redundancy requirements is found.

[0042] First, the specific implementation of query vector processing and target data table matching is introduced. The input query text is segmented and semantically parsed to extract entity information and relationship information. Entity information includes specific objects such as table names and field names, and relationship information includes operation types such as join and group by. Entity information is converted into 300-dimensional entity vectors, and relationship information is converted into 100-dimensional relationship vectors. In practice, the weight of the entity vector can be set to 0.7 and the weight of the relationship vector can be set to 0.3, and the final target query vector can be generated through weighted combination.

[0043] Let's take a specific example to illustrate. Suppose the query content is "query the association between user order information and payment records". Through word segmentation, we can get entity words such as "user", "order", "payment", etc., and relation words such as "association". After converting these words into vectors and performing weighted combination, we can get the target query vector.

[0044] Next, the relevance between the target query vector and each data table in the knowledge base is calculated. The relevance calculation considers three aspects: entity matching, relationship matching, and field coverage. Entity matching reflects the similarity between table names and field names; relationship matching reflects the matching degree of association types between tables; field coverage indicates the coverage of the fields required for the query in the target table. The weights of these three items are set to 0.4, 0.3, and 0.3 respectively. When the comprehensive score exceeds 0.75, the table is included in the candidate set.

[0045] After obtaining the candidate data tables, it is necessary to analyze the structural similarity between the tables. By comparing the field sets of the two tables, the ratio of the number of common fields to the total number of fields is calculated as the structural similarity. For example, the user table and the order table may contain the user_id field in common, and their total number of fields is 10 and 8 respectively, so their structural similarity is 0.1.

[0046] Based on the structural similarity matrix, the semantic association between fields is further analyzed. The name, type, annotation and other information of each field are extracted and converted into a semantic feature vector. In practical applications, the pre-trained word vector model can be used to extract features and weight them according to the importance of the field. Finally, the association matrix reflecting the strength of the semantic association between fields is obtained.

[0047] When constructing a data lineage directed weighted graph, candidate data tables are used as nodes and relationships between tables are used as edges. The weight of an edge needs to take into account multiple factors: data dependency strength (accounting for 0.4) reflects the reference relationship between tables; data flow frequency (accounting for 0.3) indicates the frequency of data updates; and time correlation (accounting for 0.3) reflects the timeliness of data. The edge weight is obtained by weighted calculation of these factors.

[0048] When searching for the optimal association path, the cost of each possible path is first calculated. The path cost takes into account the path length, edge weight, and reliability coefficient. The reliability coefficient is derived from the historical data quality assessment and reflects the data credibility. The path with the smallest cost is selected as the candidate association path.

[0049] The quality of the candidate association paths is evaluated, and their quality scores and redundancy are calculated. The quality score reflects the availability of the path, which is measured by the balance between the cumulative effect of the edge weights in the path and the connection overhead. The redundancy represents the proportion of redundant connections in the path, which is determined by comparing the size of the required edge set and the actual edge set.

[0050] When the quality score exceeds 0.8 and the redundancy is less than 0.2, it can be determined as the final association path. Otherwise, it is necessary to select the path with the second lowest cost and re-evaluate until a path that meets the conditions is found.

[0051] This application can achieve: The accuracy of data table association is improved through vectorization processing and multi-dimensional matching, which can accurately identify query intent and find the most relevant data table. The use of structural similarity and semantic association analysis realizes the deep mining of relationships between data tables, avoiding misjudgment caused by relying solely on surface features. The introduction of quality assessment and redundancy control mechanisms ensures that the generated association paths not only meet business needs but also have high execution efficiency, improving query performance and resource utilization.

[0052] In an optional implementation, according to the index information in the metadata information, index matching is performed on the data screening conditions and table connection conditions involved, and the query method with the highest index coverage is selected as the index matching result; based on the table association order and the index matching result, a complete SQL query statement including the data screening conditions, multi-table connection relationships and target fields is generated, including: Obtain the index information and query condition set of the data table, calculate the matching degree between each index and the query condition, calculate the index usage cost according to the index storage overhead, access time and maintenance cost, and select the index combination with the minimum index usage cost and index coverage exceeding the preset coverage threshold as the candidate index set; Analyze the query conditions based on the candidate index set, calculate the selectivity score of the data screening conditions, and determine the selectivity score according to the ratio of the amount of data that meets the screening conditions to the total amount of data in the table; evaluate the table connection conditions, calculate the connection cost value based on the table size involved in the connection, the connection index cardinality and the connection type, and select the connection condition combination whose connection cost value is lower than the preset cost threshold; Using the connection condition combination to construct a multi-table connection tree, using the connection cost value as the weight of the connection edge, calculating the size of the intermediate result set of different connection sequences, sorting all connection sequences based on the selectivity and calculation cost of the connection steps, and selecting the sequence with the smallest total cost as the optimal connection sequence; Generate an initial SQL query statement according to the optimal connection sequence and the candidate index set, and calculate the query integrity score of the query statement, wherein the integrity score includes three dimensions: query field coverage, connection condition integrity, and index utilization; at the same time, build a performance prediction model based on theoretical execution time and IO overhead to calculate the expected execution efficiency of the query statement; When the query integrity score is lower than a preset score threshold or the expected execution efficiency is lower than a preset efficiency threshold, the adaptive tuning process is started by adjusting the connection sequence, updating the index selection, and optimizing the query structure until the performance requirements are met or the maximum number of tuning times is reached.

[0053] The present invention provides a SQL query generation method based on index matching and table connection optimization. The method first obtains metadata information of a data table, including index information and a query condition set. Then, the matching degree between each index and the query condition is calculated, and the index usage cost is calculated by considering the storage overhead, access time and maintenance cost of the index.

[0054] During the index matching process, the system assigns a matching score to each index. For example, suppose there is a user table (user_table) containing the fields id, name, age, and city. The table has two indexes: idx_name_age(name,age) and idx_city(city). If the query condition is "WHERE name = 'John' AND age>30", the matching score of the idx_name_age index will be higher because it completely covers the query condition.

[0055] Next, the system calculates the cost of each index. This takes into account the index's storage space, access speed, and maintenance difficulty. For example, for the idx_name_age index, if it takes up a large storage space (such as 1GB), but can significantly improve query speed (such as reducing query time from 5 seconds to 0.1 seconds), and has a moderate maintenance cost, then its cost may be evaluated as medium.

[0056] The system will select the index combination with the lowest index usage cost and index coverage exceeding the preset coverage threshold as the candidate index set. Assuming the preset coverage threshold is 80%, in the above example, the idx_name_age index may be selected as a candidate index because it completely covers the query condition.

[0057] Based on the candidate index set, the system will conduct an in-depth analysis of the query conditions. First, the selectivity score of the data filter condition is calculated. The selectivity score is determined by the ratio of the amount of data that meets the filter condition to the total amount of data in the table. For example, if user_table has a total of 1 million records, and there are 5,000 records that meet the condition "name = 'John' AND age>30", then the selectivity score of this filter condition is 0.005 (5000 / 1000000).

[0058] For table join conditions, the system will calculate the join cost based on the size of the tables involved in the join, the cardinality of the join index, and the join type. Suppose there is also an order table (order_table) that needs to be joined with user_table. If user_table has 1 million records and order_table has 10 million records, and the join condition is user_table.id = order_table.user_id, and there is an index on user_id, then the join cost may be low. The system will filter out join condition combinations whose join cost is lower than the preset cost threshold.

[0059] Using the selected connection condition combinations, the system will build a multi-table connection tree. In this connection tree, each connection operation is represented as an edge, and the weight of the edge is the previously calculated connection cost. The system will calculate the size of the intermediate result set of different connection sequences, and sort all possible connection sequences based on the selectivity and computational cost of each connection step. Finally, the system will select the sequence with the smallest total cost as the optimal connection sequence.

[0060] For example, if there are three tables, user_table, order_table, and product_table, that need to be joined, the system may try the following join sequence: 1. (user_table JOIN order_table) JOIN product_table; 2. (order_table JOIN product_table) JOIN user_table; 3. (user_table JOIN product_table) JOIN order_table; Assuming that after calculation, the total cost of the first connection sequence is the smallest, it will be selected as the optimal connection sequence.

[0061] Based on the optimal connection sequence and candidate index set, the system will generate an initial SQL query statement. Then, the system will calculate the query completeness score of this query statement. The completeness score includes three dimensions: query field coverage, connection condition completeness, and index utilization.

[0062] The system checks whether the query covers all required fields, contains all necessary join conditions, and makes full use of available indexes.

[0063] At the same time, the system will build a performance prediction model based on theoretical execution time and IO overhead to calculate the expected execution efficiency of the query statement. This prediction model may take into account factors such as the database's hardware configuration, current load, and historical query performance.

[0064] If the query completeness score is lower than the preset score threshold (for example, 0.8) or the expected execution efficiency is lower than the preset efficiency threshold (for example, the expected execution time exceeds 5 seconds), the system will start the adaptive tuning process. During this process, the system may try the following optimization strategies: 1. Adjust the connection sequence: for example, change the original (user_table JOIN order_table) JOINproduct_table to (order_table JOIN product_table) JOIN user_table.

[0065] 2. Update index selection: For example, if you find that there is no index on the order_date field, you may be advised to create a new index to improve query efficiency.

[0066] 3. Optimize the query structure: for example, rewrite the subquery into JOIN, or split complex conditions into multiple simple conditions.

[0067] The system will repeat this optimization process until the performance requirements are met or the maximum number of tuning times (such as 10) is reached. After each optimization, the system will re-evaluate the query's completeness score and expected execution efficiency to determine whether further optimization is needed.

[0068] Through this method, the system can generate SQL query statements that both meet query requirements and have high execution efficiency.

[0069] This application can achieve: By comprehensively analyzing index information and query conditions, it is possible to select the optimal index combination and significantly improve query efficiency. It not only considers the degree of match between the index and the query conditions, but also weighs the index storage overhead, access time, and maintenance cost, thus achieving a good balance between query performance and system resource utilization.

[0070] By building a multi-table join tree and evaluating different join sequences, the optimal table join order can be found. This method takes into account the size of the table, the selectivity of the join conditions, and the size of the intermediate result set, effectively reducing the amount of data processing during the query process, thereby significantly improving the execution efficiency of complex queries.

[0071] An adaptive tuning mechanism is introduced to dynamically optimize query statements based on query integrity scores and expected execution efficiency. This continuous optimization method can not only cope with complex query scenarios, but also adapt to changes in the database environment, ensuring the long-term stability of query performance. In this way, the method can generate efficient SQL query statements in various complex database environments, greatly improving the overall performance of the database and user experience.

[0072] In an optional implementation, the connection condition combination is used to construct a multi-table connection tree, the connection cost value is used as the weight of the connection edge, the size of the intermediate result set of different connection sequences is calculated, all possible connection sequences are sorted in combination with the selectivity and calculation cost of the connection step, and the sequence with the smallest total cost is selected as the optimal connection sequence, which includes: A connection relevance evaluation model is constructed based on the number of foreign key associations and data relevance between data tables to calculate the connection relevance score between any two data tables. The connection relevance score is determined by the ratio of the number of foreign key associations to the number of table records and the data relevance coefficient. Calculate the weight value of the connection edge according to the connection relevance score, the weight value is obtained by weighted calculation of the connection operation cost, input and output overhead and processor calculation overhead, and use the weight value as the weight attribute of the connection tree edge; Calculate the cumulative execution cost for all connection sequences, the cumulative execution cost includes the connection operation cost, the intermediate result transmission cost and the storage cost, and calculate the sequence execution efficiency score by combining the selection rate and processing cost of the connection steps; Selecting a sequence with the highest execution efficiency score from all connection sequences as a candidate sequence, and calculating an expected resource consumption value of the candidate sequence, wherein the expected resource consumption value includes a processor usage rate, a memory occupancy rate, and a network bandwidth usage rate; When the expected value of resource consumption exceeds the system capacity threshold, the sequence optimization process is triggered to reduce resource consumption by adjusting the connection order, splitting the connection steps or parallel processing until the optimal connection sequence that meets the system resource constraints is found.

[0073] For example, in the optimization of multi-table connection queries in a database, it is first necessary to build a table connection correlation evaluation model. By analyzing the foreign key associations between data tables, the number of foreign keys between each pair of tables is counted, and the ratio to the total number of records in the table is calculated. At the same time, combined with the field value distribution characteristics, the correlation coefficient between fields is calculated using data sampling. Taking the order table and the user table as an example, assuming that the order table has 10 million records and the user table has 1 million records, the two tables are associated by user ID, and the correlation coefficient is 0.8, a higher connection correlation score can be obtained.

[0074] After obtaining the connection relevance score, the weight value of the connection edge is further calculated. Weight calculation needs to consider multiple factors: the CPU overhead of the connection operation itself, the IO overhead of data reading, and the storage overhead of the intermediate results. Taking the connection of two large tables as an example, if the input table sizes are 10GB and 20GB respectively, and indexes are built on the connection fields, the CPU overhead of the connection operation is relatively small, and the main overhead comes from data reading. Based on this, a reasonable weight value can be calculated.

[0075] After building the connection tree based on the above weight values, it is necessary to evaluate the execution efficiency of different connection sequences. For each possible connection sequence, the total execution cost is calculated cumulatively. Taking the three-table connection as an example, assuming that the sizes of tables A, B, and C are 1GB, 2GB, and 5GB respectively, the intermediate result set size and transmission cost of the sequences "A→B→C" and "B→C→A" will be significantly different. By simulating the selectivity of each connection step (for example, 0.1 means that the result set is only 10% of the input data volume), the execution efficiency of different sequences can be accurately evaluated.

[0076] After obtaining the candidate sequence with the highest execution efficiency score, its resource consumption needs to be estimated. By analyzing historical execution data, a resource consumption prediction model is established. For example, for a connection operation with a total data volume of 50GB, it is estimated that 40% of CPU resources, 30GB of memory space, and 2Gbps of network bandwidth will be required. If the predicted value exceeds the system threshold (for example, the memory usage rate must not exceed 80%), the connection sequence needs to be optimized.

[0077] Optimization methods include adjusting the connection order, splitting large connection steps into multiple small steps, or introducing parallel processing mechanisms. Taking high memory usage as an example, a large connection operation can be split into multiple small batches for execution, with each batch processing part of the data, thereby controlling the memory peak within an acceptable range. Through iterative optimization, the optimal connection sequence that meets performance requirements and does not exceed resource constraints is finally obtained.

[0078] This application can achieve: By building an accurate connection relevance evaluation model and a comprehensive cost calculation system, the execution efficiency of multi-table connection queries has been significantly improved, unnecessary resource waste has been reduced, and query performance has been greatly optimized. Based on resource consumption prediction and dynamic optimization mechanisms, the system can adaptively adjust connection strategies, effectively avoid resource overload problems, improve system stability and reliability, and ensure smooth operation in high-concurrency scenarios. The flexible connection sequence optimization solution, combined with technical means such as parallel processing and batch execution, achieves efficient processing of large-scale data, enhances the scalability and adaptability of the system, and can meet the needs of data processing of different scales.

[0079] Figure 3 FIG. 1 is a schematic diagram of the structure of a natural language SQL conversion system based on a data platform and a large language model according to an embodiment of the present invention. Figure 3 As shown, the system comprises: The first unit is used to convert the acquired metadata information, data lineage information, business indicator information and historical query information into vector form through the data service API and store them in the knowledge vector library; use natural language processing technology to identify the intent of the query request input by the user, and convert the extracted intent information into a query vector; perform similarity matching on the query vector with the metadata vector, lineage vector, indicator vector and historical query vector in the knowledge vector library; determine the data table, field and association relationship to be queried based on the matching results; when the matching results show that the query conditions are incomplete, generate a supplementary query according to the business indicator information in the knowledge vector library; obtain user feedback information on the supplementary query and update the query vector; repeat the similarity matching process until the complete query conditions are obtained; The second unit is used to extract the structural information and field definition of the target data table from the knowledge vector library based on the query vector and the matching result; construct an initial association path according to the data flow path in the data lineage information; retrieve historical query records containing the same data table combination from the historical query information based on the data table in the initial association path; sort the historical query records according to the query execution time, and select the table connection order in the historical query record with the shortest execution time as the optimized table association order; The third unit is used to perform index matching on the data screening conditions and table connection conditions involved according to the index information in the metadata information, and select the query method with the highest index coverage as the index matching result; based on the table association order and the index matching result, generate a complete SQL query statement including the data screening conditions, multi-table connection relationships and target fields; execute the SQL query statement and record the query results; and update the SQL query statement, the query vector, the query execution time, the associated path and the query results to the historical query information.

[0080] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0081] A fourth aspect of the embodiments of the present invention is: A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0082] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A natural language SQL conversion method based on a data platform and a large language model, characterized in that: include: The acquired metadata information, data lineage information, business indicator information and historical query information are converted into vector form through the data service API and stored in the knowledge vector library; Using natural language processing technology to identify the intent of the query request input by the user, converting the extracted intent information into a query vector; performing similarity matching on the query vector with the metadata vector, lineage vector, indicator vector and historical query vector in the knowledge vector library; determining the data table, field and association relationship to be queried based on the matching result; when the matching result shows that the query condition is incomplete, generating a supplementary query based on the business indicator information in the knowledge vector library; obtaining user feedback information on the supplementary query, and updating the query vector; repeating the similarity matching process until the complete query condition is obtained; Based on the query vector and the matching result, the structural information and field definition of the target data table are extracted from the knowledge vector library; an initial association path is constructed according to the data flow path in the data lineage information; based on the data table in the initial association path, historical query records containing the same data table combination are retrieved from the historical query information; the historical query records are sorted according to the query execution time, and the table connection sequence in the historical query record with the shortest execution time is selected as the optimized table association sequence; According to the index information in the metadata information, index matching is performed on the data screening conditions and table connection conditions involved, and the query method with the highest index coverage rate is selected as the index matching result; Based on the table association order and the index matching result, a complete SQL query statement including data screening conditions, multi-table connection relationships and target fields is generated; the SQL query statement is executed and the query result is recorded; The SQL query statement, the query vector, the query execution time, the associated path and the query result are updated to the historical query information.

2. The method according to claim 1, characterized in that The metadata information includes data table structure, field definition, index information, constraints, data types and field comments; The data lineage information includes data flow paths, upstream and downstream associations of data objects, and task dependencies; The business indicator information includes sales statistics, conversion rate calculation and inventory turnover rate; The historical query information includes SQL query statements, query frequency and query execution time.

3. The method according to claim 1, characterized in that Determine the data table, field and associated relationship to be queried based on the matching results; when the matching results show that the query conditions are incomplete, generate a supplementary query based on the business indicator information in the knowledge vector library; Acquiring user feedback information on the supplementary query and updating the query vector includes: The query vector is matched with the vectors in the knowledge vector library by using a similarity calculation method. The query vector and the vectors in the knowledge vector library are matched in multiple dimensions by cosine similarity calculation, and the matching result is determined according to a set similarity threshold; Based on the matching results, a query graph is constructed, wherein the query graph includes a vertex set representing a data table and an edge set representing an association relationship between tables, and a connectivity index of the query graph is calculated, wherein the connectivity index represents the integrity of the graph through the relationship between the number of edges and the number of vertices; when the connectivity index is less than a preset connectivity threshold, a data table node that needs to supplement the association condition is marked; Verify the validity of the fields involved in the query, and calculate the field validity weight based on the field relevance, query frequency and null value rate. The field relevance represents the degree of association between the field and the business indicator, the query frequency represents the frequency of use of the field in historical queries, and the null value rate represents the data quality of the field; Based on the field validity weight, a hierarchical query strategy is constructed, wherein the hierarchical query strategy includes a core business condition confirmation layer, an association relationship supplement layer, and a query constraint refinement layer, and each layer is assigned a different importance coefficient; the query priority is calculated according to the influence of the business indicator on the supplementary condition, and the corresponding supplementary query is generated according to the query priority; Obtain user feedback information on the supplementary query, calculate vector update strength based on the feedback information, where the vector update strength is determined by the number of feedbacks, the feedback importance, and the learning rate; enhance and normalize the query vector according to the vector update strength to generate an updated query vector.

4. The method according to claim 1, characterized in that Based on the query vector and the matching result, extracting the structure information and field definition of the target data table from the knowledge vector library; constructing the initial association path according to the data flow path in the data lineage information includes: Decomposing the query vector into an entity vector and a relationship vector, and performing a linear combination of the entity vector and the relationship vector based on a preset weight coefficient to generate a target query vector; calculating the relevance score between the target query vector and each data table in the knowledge vector library, wherein the relevance score is obtained by weighted calculation of the entity matching score, the relationship matching score, and the field coverage, and screening the data table whose relevance score is greater than the relevance threshold as a candidate data table; Constructing a structural similarity matrix for the candidate data tables, wherein each element value in the structural similarity matrix is ​​obtained by calculating the ratio of the intersection and the union of the field sets of the corresponding two data tables; calculating a semantic feature vector for each field based on the structural similarity matrix, performing weighted summation on the semantic feature vectors to obtain a field semantic vector, and calculating the field semantic association between the candidate data tables based on the field semantic vectors; A data lineage directed weighted graph is constructed based on the semantic association of the fields, wherein the nodes of the directed weighted graph represent candidate data tables, the edges represent associations between tables, and the edge weights are obtained by weighted calculation of data dependency strength, data flow frequency, and time correlation; and the path cost values ​​of all association paths in the graph are calculated based on the edge weights and the pre-calculated reliability coefficients; Selecting a path with the smallest path cost from the association paths as a candidate association path, calculating a quality score of the candidate association path, wherein the quality score is obtained by the ratio of the cumulative multiplication of all edge weight values ​​in the path to the sum of squares of connection costs; and calculating the redundancy of the candidate association path, wherein the redundancy is determined based on the ratio of a required edge set to maintain the association relationship of the data table to an actual edge set; When the quality score is greater than a preset quality threshold and the redundancy is less than a preset redundancy threshold, the candidate association path is determined as the final association path; otherwise, a path with the second smallest path cost is selected from the remaining candidate association paths as a new candidate association path, and the quality score calculation and redundancy calculation are repeatedly performed until a final association path that meets the quality and redundancy requirements is found.

5. The method according to claim 1, characterized in that According to the index information in the metadata information, index matching is performed on the data screening conditions and table connection conditions involved, and the query method with the highest index coverage rate is selected as the index matching result; Based on the table association order and the index matching result, a complete SQL query statement including data screening conditions, multi-table connection relationships and target fields is generated, including: Obtain the index information and query condition set of the data table, calculate the matching degree between each index and the query condition, calculate the index usage cost according to the index storage overhead, access time and maintenance cost, and select the index combination with the minimum index usage cost and index coverage exceeding the preset coverage threshold as the candidate index set; Analyze the query conditions based on the candidate index set, calculate the selectivity score of the data screening conditions, and determine the selectivity score according to the ratio of the amount of data that meets the screening conditions to the total amount of data in the table; evaluate the table connection conditions, calculate the connection cost value based on the table size involved in the connection, the connection index cardinality and the connection type, and select the connection condition combination whose connection cost value is lower than the preset cost threshold; Using the connection condition combination to construct a multi-table connection tree, using the connection cost value as the weight of the connection edge, calculating the size of the intermediate result set of different connection sequences, sorting all connection sequences based on the selectivity and calculation cost of the connection steps, and selecting the sequence with the smallest total cost as the optimal connection sequence; Generate an initial SQL query statement according to the optimal connection sequence and the candidate index set, and calculate the query integrity score of the query statement, wherein the integrity score includes three dimensions: query field coverage, connection condition integrity, and index utilization; at the same time, build a performance prediction model based on theoretical execution time and IO overhead to calculate the expected execution efficiency of the query statement; When the query integrity score is lower than a preset score threshold or the expected execution efficiency is lower than a preset efficiency threshold, the adaptive tuning process is started by adjusting the connection sequence, updating the index selection, and optimizing the query structure until the performance requirements are met or the maximum number of tuning times is reached.

6. The method according to claim 1, characterized in that Using the connection condition combination to construct a multi-table connection tree, using the connection cost value as the weight of the connection edge, calculating the size of the intermediate result set of different connection sequences, sorting all possible connection sequences based on the selectivity and calculation cost of the connection steps, and selecting the sequence with the smallest total cost as the optimal connection sequence includes: A connection relevance evaluation model is constructed based on the number of foreign key associations and data relevance between data tables to calculate the connection relevance score between any two data tables. The connection relevance score is determined by the ratio of the number of foreign key associations to the number of table records and the data relevance coefficient. Calculate the weight value of the connection edge according to the connection relevance score, the weight value is obtained by weighted calculation of the connection operation cost, input and output overhead and processor calculation overhead, and use the weight value as the weight attribute of the connection tree edge; Calculate the cumulative execution cost for all connection sequences, the cumulative execution cost includes the connection operation cost, the intermediate result transmission cost and the storage cost, and calculate the sequence execution efficiency score by combining the selection rate and processing cost of the connection steps; Selecting a sequence with the highest execution efficiency score from all connection sequences as a candidate sequence, and calculating an expected resource consumption value of the candidate sequence, wherein the expected resource consumption value includes a processor usage rate, a memory occupancy rate, and a network bandwidth usage rate; When the expected value of resource consumption exceeds the system capacity threshold, the sequence optimization process is triggered to reduce resource consumption by adjusting the connection order, splitting the connection steps or parallel processing until the optimal connection sequence that meets the system resource constraints is found.

7. A natural language SQL conversion system based on a data platform and a large language model, used to implement the method as claimed in any one of claims 1 to 6, characterized in that: include: The first unit is used to convert the acquired metadata information, data lineage information, business indicator information and historical query information into vector form through the data service API and store it in the knowledge vector library; Using natural language processing technology to identify the intent of the query request input by the user, converting the extracted intent information into a query vector; performing similarity matching on the query vector with the metadata vector, lineage vector, indicator vector and historical query vector in the knowledge vector library; determining the data table, field and association relationship to be queried based on the matching result; when the matching result shows that the query condition is incomplete, generating a supplementary query based on the business indicator information in the knowledge vector library; obtaining user feedback information on the supplementary query, and updating the query vector; repeating the similarity matching process until the complete query condition is obtained; The second unit is used to extract the structural information and field definition of the target data table from the knowledge vector library based on the query vector and the matching result; construct an initial association path according to the data flow path in the data lineage information; retrieve historical query records containing the same data table combination from the historical query information based on the data table in the initial association path; sort the historical query records according to the query execution time, and select the table connection order in the historical query record with the shortest execution time as the optimized table association order; The third unit is used to perform index matching on the data screening conditions and table connection conditions involved according to the index information in the metadata information, and select the query method with the highest index coverage as the index matching result; Based on the table association order and the index matching result, a complete SQL query statement including data screening conditions, multi-table connection relationships and target fields is generated; the SQL query statement is executed and the query result is recorded; The SQL query statement, the query vector, the query execution time, the associated path and the query result are updated to the historical query information.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images