Natural Language to SQL Conversion Method Based on Data Platform and Large Language Model

Abstract

The present invention provides a natural language SQL conversion method based on a data platform and a large language model, which relates to the technical field of data processing. It includes vectorizing and storing metadata, lineage, metrics, and historical query information, discriminating the intent of a user query request and performing vector matching, determining query conditions based on the matching results, constructing an association path according to data lineage, optimizing the table join order using historical queries, and generating an optimal SQL statement in combination with index matching. The present invention can achieve intelligent conversion from natural language to efficient SQL, improve query efficiency, and lower the user's usage threshold.

Description

Technical Field

[0001] The present invention relates to data processing technologies, and in particular to a natural language SQL conversion method based on a data platform and a large language model. Background Art

[0002] With the rapid development of big data technologies, enterprises have accumulated a large amount of data assets. To make full use of these data resources for analysis and decision-making, data analysts need to frequently write SQL query statements to obtain the required data. However, for business personnel who are not familiar with SQL syntax, directly writing SQL query statements is quite difficult. Although there are some natural language to SQL tools on the market currently, these tools can often only handle simple single-table query scenarios and still face many challenges when dealing with complex multi-table association queries.

[0003] The existing technologies mainly have the following deficiencies:

[0004] Existing natural language to SQL tools lack an in-depth understanding of business semantics and cannot accurately identify the business rules and metric definitions implicit in the user's query intent, resulting in a deviation between the generated SQL query results and the user's actual needs.

[0005] When dealing with multi-table association queries, existing technologies usually adopt a fixed table connection strategy and fail to make full use of historical query experience and database index information to optimize query performance, resulting in low query execution efficiency.

[0006] When dealing with incomplete or ambiguous query requests, existing technologies lack an effective interaction mechanism to clarify and supplement query conditions, and cannot guarantee the accuracy and integrity of the generated SQL query statements, affecting the usability of query results. Summary of the Invention

[0007] Embodiments of the present invention provide a natural language SQL conversion method based on a data platform and a large language model, which can solve the problems in the existing technologies.

[0008] In the first aspect of the embodiments of the present invention,

[0009] A natural language SQL conversion method based on a data platform and a large language model is provided, including:

[0010] The obtained metadata information, data lineage information, business metric information, and historical query information are converted into vector form through a data service API and stored in a knowledge vector library; natural language processing technology is used to discriminate the intent of the query request input by the user, and the extracted intent information is converted into a query vector; the query vector is matched with the metadata vector, lineage vector, metric vector, and historical query vector in the knowledge vector library for similarity; based on the matching result, the data table, fields, and association relationships to be queried are determined; when the matching result shows that the query conditions are incomplete, a supplementary query is generated according to the business metric information in the knowledge vector library; the feedback information of the user on the supplementary query is obtained, and the query vector is updated; the similarity matching process is repeatedly executed until complete query conditions are obtained;

[0011] Based on the query vector and the matching result, the structure information and field definitions of the target data table are extracted from the knowledge vector library; an initial association path is constructed according to the data flow path in the data lineage information; based on the data tables in the initial association path, historical query records containing the same data table combination are retrieved from the historical query information; the historical query records are sorted according to the query execution time, and the table join order in the historical query record with the shortest execution time is selected as the optimized table association order;

[0012] According to the index information in the metadata information, index matching is performed on the involved data filtering conditions and table join conditions, and the query method with the highest index coverage rate is selected as the index matching result; based on the table association order and the index matching result, a complete SQL query statement containing data filtering conditions, multi-table join relationships, and target fields is generated; the SQL query statement is executed and the query result is recorded; the SQL query statement, the query vector, the query execution time, the association path, and the query result are updated to the historical query information.

[0013] The metadata information includes data table structure, field definitions, index information, constraint conditions, data types, and field comments;

[0014] The data lineage information includes data flow path, upstream and downstream association relationships of data objects, and task dependencies;

[0015] The business metric information includes sales statistics, conversion rate calculation, and inventory turnover rate;

[0016] The historical query information includes SQL query statements, query frequencies, and query execution times.

[0017] Determine the data table, fields, and association relationships to be queried based on the matching results; when the matching results show that the query conditions are incomplete, generate supplementary inquiries according to the business metric information in the knowledge vector library; obtain the feedback information of the user on the supplementary inquiries, and update the query vector, including:

[0018] Use a similarity calculation method to match the query vector with the vectors in the knowledge vector library. The query vector and the vectors in the knowledge vector library are multi-dimensionally matched through cosine similarity calculation, and the matching results are determined according to the set similarity threshold.

[0019] Construct a query graph based on the matching results. The query graph includes a vertex set representing data tables and an edge set representing the inter-table association relationships. Calculate the connectivity index of the query graph. The connectivity index characterizes the integrity of the graph through the relationship between the number of edges and the number of vertices. When the connectivity index is less than the preset connectivity threshold, mark the data table nodes that need to supplement the association conditions.

[0020] Conduct validity verification on the fields involved in the query, and calculate the field validity weight according to the field correlation degree, query frequency, and null value rate. The field correlation degree characterizes the association degree between the field and the business metric, the query frequency characterizes the usage frequency of the field in historical queries, and the null value rate characterizes the data quality of the field.

[0021] Construct a hierarchical inquiry strategy based on the field validity weight. The hierarchical inquiry strategy includes a core business condition confirmation layer, an association relationship supplement layer, and a query constraint refinement layer, and different importance coefficients are assigned to each layer. Calculate the inquiry priority according to the influence degree of the business metric on the conditions to be supplemented, and generate corresponding supplementary inquiries according to the inquiry priority.

[0022] Obtain the feedback information of the user on the supplementary inquiries, calculate the vector update intensity based on the feedback information. The vector update intensity is jointly determined by the number of feedbacks, the importance degree of the feedback, and the learning rate. Enhance and normalize the query vector according to the vector update intensity to generate an updated query vector.

[0023] Extract the structure information and field definitions of the target data table from the knowledge vector library based on the query vector and the matching results; construct an initial association path according to the data transfer path in the data lineage information, including:

[0024] Decompose the query vector into an entity vector and a relationship vector, and perform a linear combination of the entity vector and the relationship vector based on a preset weight coefficient to generate a target query vector; calculate the relevance score between the target query vector and each data table in the knowledge vector library, where the relevance score is obtained by weighted calculation of the entity matching score, the relationship matching score, and the field coverage rate, and filter out the data tables with a relevance score greater than the relevance threshold as candidate data tables;

[0025] Construct a structural similarity matrix for the candidate data tables, where each element value in the structural similarity matrix is obtained by calculating the ratio of the intersection to the union of the field sets of the corresponding two data tables; calculate the semantic feature vector of each field based on the structural similarity matrix, and perform a weighted sum of the semantic feature vectors to obtain the field semantic vector, and calculate the field semantic association degree between the candidate data tables according to the field semantic vector;

[0026] Construct a data lineage directed weighted graph based on the field semantic association degree, where the nodes of the directed weighted graph represent candidate data tables, the edges represent the inter-table association relationships, and the edge weight values are obtained by weighted calculation of the data dependence strength, the data transfer frequency, and the time correlation; calculate the path cost value of all association paths in the graph according to the edge weight value and the reliability coefficient calculated in advance;

[0027] Select the path with the smallest path cost value from the association paths as the candidate association path, and calculate the quality score of the candidate association path, where the quality score is obtained by the ratio of the product of all edge weight values in the path to the sum of the squares of the connection overheads; at the same time, calculate the redundancy of the candidate association path, where the redundancy is determined based on the ratio of the necessary edge set required to maintain the data table association relationship to the actual edge set;

[0028] When the quality score is greater than the preset quality threshold and the redundancy is less than the preset redundancy threshold, determine the candidate association path as the final association path; otherwise, select the path with the second smallest path cost value from the remaining candidate association paths as the new candidate association path, and repeat the quality score calculation and redundancy calculation until the final association path that meets the quality and redundancy requirements is found.

[0029] According to the index information in the metadata information, perform index matching on the involved data filtering conditions and table connection conditions, and select the query method with the highest index coverage rate as the index matching result; based on the table association order and the index matching result, generate a complete SQL query statement including data filtering conditions, multi-table connection relationships, and target fields:

[0030] Obtain the index information and query condition set of the data table, calculate the matching degree between each index and the query conditions, calculate the index usage cost according to the storage cost, access time and maintenance cost of the index, and select the index combination with the minimum index usage cost and an index coverage rate exceeding the preset coverage threshold as the candidate index set;

[0031] Analyze the query conditions based on the candidate index set, calculate the selectivity score of the data filtering conditions, and the selectivity score is determined according to the ratio of the amount of data meeting the filtering conditions to the total amount of data in the table; evaluate the table join conditions, calculate the join cost value based on the sizes of the tables participating in the join, the join index cardinality and the join type, and filter the join condition combinations with join cost values lower than the preset cost threshold;

[0032] Use the join condition combinations to construct a multi-table join tree, use the join cost value as the weight of the join edge, calculate the sizes of the intermediate result sets of different join sequences, and sort all the join sequences in combination with the selection rate and calculation cost of the join steps, and select the sequence with the minimum total cost as the optimal join sequence;

[0033] Generate an initial SQL query statement according to the optimal join sequence and the candidate index set, calculate the query integrity score of this query statement, and the integrity score includes three dimensions: query field coverage rate, join condition integrity and index utilization rate; at the same time, construct a performance prediction model based on the theoretical execution time and IO overhead, and calculate the expected execution efficiency of the query statement;

[0034] When the query integrity score is lower than the preset score threshold or the expected execution efficiency is lower than the preset efficiency threshold, start the adaptive tuning process by adjusting the join sequence, updating the index selection, and optimizing the query structure until the performance requirements are met or the maximum number of tuning times is reached.

[0035] Using the join condition combinations to construct a multi-table join tree, using the join cost value as the weight of the join edge, calculating the sizes of the intermediate result sets of different join sequences, and sorting all possible join sequences in combination with the selection rate and calculation cost of the join steps, and selecting the sequence with the minimum total cost as the optimal join sequence includes:

[0036] Construct a join correlation evaluation model based on the number of foreign key associations and data correlation between data tables, calculate the join correlation scores between any two data tables, and the join correlation scores are jointly determined by the ratio of the number of foreign key associations to the number of table records and the data correlation coefficient;

[0037] Calculate the weight value of the join edge according to the join correlation score, and the weight value is obtained by weighted calculation of the join operation cost, input / output overhead and processor calculation overhead, and use the weight value as the weight attribute of the join tree edge;

[0038] Calculate the cumulative execution cost for all connection sequences. The cumulative execution cost includes the connection operation cost, the intermediate result transmission cost, and the storage cost. Combine the selectivity and processing cost of the connection step to calculate the sequence execution efficiency score.

[0039] Select the sequence with the highest execution efficiency score from all connection sequences as the candidate sequence, and calculate the expected value of resource consumption of the candidate sequence. The expected value of resource consumption includes the processor utilization rate, the memory occupancy rate, and the network bandwidth utilization rate.

[0040] When the expected value of resource consumption exceeds the system capacity threshold, trigger the sequence optimization process, and reduce the resource consumption by adjusting the connection order, splitting the connection step, or parallel processing until the optimal connection sequence that meets the system resource constraints is found.

[0041] In the second aspect of the embodiments of the present invention, a natural language SQL conversion system based on a data platform and a large language model is provided, including:

[0042] The first unit is used to convert the obtained metadata information, data lineage information, business metric information, and historical query information into vector form through the data service API and store them in the knowledge vector library; use natural language processing technology to discriminate the intent of the query request input by the user, and convert the extracted intent information into a query vector; perform similarity matching between the query vector and the metadata vector, lineage vector, metric vector, and historical query vector in the knowledge vector library; determine the data tables, fields, and association relationships to be queried based on the matching result; when the matching result shows that the query conditions are incomplete, generate a supplementary query according to the business metric information in the knowledge vector library; obtain the feedback information of the user on the supplementary query, and update the query vector; repeat the similarity matching process until complete query conditions are obtained.

[0043] The second unit is used to extract the structure information and field definitions of the target data table from the knowledge vector library based on the query vector and the matching result; construct an initial association path according to the data flow path in the data lineage information; retrieve historical query records containing the same data table combination from the historical query information based on the data tables in the initial association path; sort the historical query records according to the query execution time, and select the table connection order in the historical query record with the shortest execution time as the optimized table association order.

[0044] A third unit is configured to perform index matching on data filtering conditions and table connection conditions involved according to index information in the metadata information, select a query method with the highest index coverage rate as the index matching result; generate a complete SQL query statement including data filtering conditions, multi-table connection relationships, and target fields based on the table association order and the index matching result; execute the SQL query statement and record the query result; update the SQL query statement, the query vector, the query execution time, the association path, and the query result to the historical query information.

[0045] In a third aspect of the embodiments of the present invention

[0046] There is provided an electronic device, including:

[0047] A processor;

[0048] A memory for storing instructions executable by the processor;

[0049] Wherein, the processor is configured to call the instructions stored in the memory to execute the method described above.

[0050] In a fourth aspect of the embodiments of the present invention,

[0051] There is provided a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described above is implemented.

[0052] The beneficial effects of the present application are as follows:

[0053] By vectorizing and storing metadata, data lineage, business metrics, and historical query information, and using natural language processing technology for intent discrimination and similarity matching, the present invention can accurately understand the user's query intent and quickly locate relevant data tables and fields. When the query conditions are incomplete, the system can also generate supplementary inquiries according to the business metric information to ensure that complete query conditions are obtained, thereby improving the query accuracy and efficiency.

[0054] The present invention constructs an initial association path using data lineage information and optimizes the table association order by analyzing historical query records, which can effectively reduce the complexity and execution time of data queries. At the same time, by matching and optimizing the index information and selecting the optimal query method, the query performance is further improved.

[0055] The present invention updates the generated SQL query statement, query vector, execution time, association path, and query result to the historical query information, forming a continuously optimized closed-loop system. This method can not only continuously improve the query efficiency, but also provide valuable references for subsequent queries, thereby realizing the continuous optimization of query performance and the continuous improvement of the system's intelligence level. Brief Description of the Drawings

[0056] Figure 1 This is a schematic flowchart of the natural language SQL conversion method based on a data platform and a large language model according to an embodiment of the present invention;

[0057] Figure 2 This is a schematic logical diagram of the natural language SQL conversion method based on a data platform and a large language model according to an embodiment of the present invention

[0058] Figure 3 This is a schematic structural diagram of the natural language SQL conversion system based on a data platform and a large language model according to an embodiment of the present invention. Detailed Description of the Embodiments

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0060] The technical solutions of the present invention will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0061] Figure 1 and Figure 2 are respectively a schematic flowchart and a schematic logical diagram of the natural language SQL conversion method based on a data platform and a large language model according to an embodiment of the present invention. The method includes:

[0062] S101. Convert the obtained metadata information, data lineage information, business metric information, and historical query information into vector form through a data service API and store them in a knowledge vector library; use natural language processing technology to discriminate the intent of the query request input by the user, and convert the extracted intent information into a query vector; perform similarity matching between the query vector and the metadata vector, lineage vector, metric vector, and historical query vector in the knowledge vector library; determine the data tables, fields, and association relationships to be queried based on the matching results; when the matching results show that the query conditions are incomplete, generate a supplementary query according to the business metric information in the knowledge vector library; obtain the feedback information of the user on the supplementary query and update the query vector; repeat the similarity matching process until complete query conditions are obtained;

[0063] S102. Based on the query vector and the matching result, extract the structure information and field definitions of the target data table from the knowledge vector library; construct an initial association path according to the data flow path in the data lineage information; based on the data tables in the initial association path, retrieve historical query records containing the same data table combination from the historical query information; sort the historical query records according to the query execution time, and select the table join order in the historical query record with the shortest execution time as the optimized table association order;

[0064] S103. According to the index information in the metadata information, perform index matching on the involved data filtering conditions and table join conditions, and select the query method with the highest index coverage rate as the index matching result; based on the table association order and the index matching result, generate a complete SQL query statement including data filtering conditions, multi-table join relationships, and target fields; execute the SQL query statement and record the query result; update the SQL query statement, the query vector, the query execution time, the association path, and the query result to the historical query information.

[0065] In an optional implementation manner, the metadata information includes data table structure, field definition, index information, constraint conditions, data type, and field annotation;

[0066] The data lineage information includes data flow path, upstream and downstream association relationships of data objects, and task dependencies;

[0067] The business metric information includes sales volume statistics, conversion rate calculation, and inventory turnover rate;

[0068] The historical query information includes SQL query statements, query frequencies, and query execution times.

[0069] First, construct a metadata collection module to obtain the table structure information of various data sources through a database connector. For relational databases, use the JDBC driver to read the data dictionary and extract basic information such as table names, field names, field types, default values, and whether they are null. At the same time, parse the DDL statements to obtain integrity constraint information such as primary key constraints, foreign key constraints, and uniqueness constraints.

[0070] When obtaining the data lineage relationship, first parse the ETL job configuration file to extract the input and output table information in the data processing tasks. By constructing a directed graph model, store the dependency relationship between tables as edges, and record the conversion rules at the same time. For stored procedures and views, identify the source tables and target tables involved by parsing their definition statements, and supplement and improve the data lineage graph.

[0071] The collection of business indicator information needs to be combined with specific business scenarios. Taking sales statistics as an example, by configuring the indicator caliber, elements such as the statistical period, aggregation dimension, and calculation rules are determined. A sales fact table is established in the data warehouse to record fields such as order amount, payment status, and refund amount. Through a pre-aggregation view, the sales amount is calculated according to different dimensions (such as date, region, category).

[0072] For the conversion rate, it is necessary to record the status changes of each link in the user behavior fact table. For example, in the e-commerce scenario, key nodes such as browsing products, adding to the shopping cart, submitting an order, and successful payment are recorded. Through behavioral sequence analysis, the conversion ratio between each link is calculated. At the same time, combined with user segmentation, the conversion performance of users with different characteristics is analyzed.

[0073] The calculation of inventory turnover rate is based on the inventory snapshot table and the inventory in and out transaction table. The daily balance quantity and amount are recorded in the inventory snapshot table, and the sales out quantity during the period is calculated through the inventory in and out transaction table. Combining the purchase in records, the timeliness of replenishment can be analyzed to optimize the inventory management strategy.

[0074] The collection of historical query information is achieved by deploying monitoring components on the database server side. Execution details such as the original SQL statement, execution timestamp, response time, and affected row count are recorded. By establishing a query pattern library, similar SQLs are classified, and the execution frequency of each type of query is counted. Index optimization suggestions are established for high-frequency queries to improve query performance.

[0075] This application can achieve:

[0076] By comprehensively collecting and analyzing metadata information, the current status of data assets can be accurately grasped, providing a basic support for data governance, improving data quality and usage efficiency. Based on data lineage analysis, the source of data anomalies can be quickly located, the scope of impact of changes can be evaluated, operation and maintenance risks can be reduced, and the reliability and maintainability of data processing can be improved. Through a standardized indicator system and historical query analysis, business decisions can be effectively supported, system performance can be optimized, user experience can be improved, and the continuous mining and application of data value can be realized.

[0077] In an alternative embodiment, the data table, fields, and association relationships to be queried are determined based on the matching result; when the matching result shows that the query condition is incomplete, a supplementary query is generated according to the business indicator information in the knowledge vector library; obtaining the feedback information of the user on the supplementary query and updating the query vector includes:

[0078] The query vector is matched with the vectors in the knowledge vector library by using a similarity calculation method. The query vector and the vectors in the knowledge vector library are multi-dimensionally matched through cosine similarity calculation, and the matching result is determined according to the set similarity threshold;

[0079] Construct a query graph based on the matching results. The query graph includes a vertex set representing data tables and an edge set representing the inter-table association relationships. Calculate the connectivity index of the query graph. The connectivity index characterizes the integrity of the graph through the relationship between the number of edges and the number of vertices. When the connectivity index is less than a preset connectivity threshold, mark the data table nodes that need to supplement association conditions.

[0080] Perform validity verification on the fields involved in the query. Calculate the field validity weight according to the field association degree, query frequency, and null value rate. The field association degree characterizes the degree of association between the field and the business indicator. The query frequency characterizes the usage frequency of the field in historical queries. The null value rate characterizes the data quality of the field.

[0081] Construct a hierarchical interrogation strategy based on the field validity weight. The hierarchical interrogation strategy includes a core business condition confirmation layer, an association relationship supplement layer, and a query constraint refinement layer. Each layer is assigned a different importance coefficient. Calculate the interrogation priority according to the influence degree of the business indicator on the conditions to be supplemented, and generate corresponding supplementary interrogations according to the interrogation priority.

[0082] Obtain the feedback information of the user on the supplementary interrogations. Calculate the vector update strength based on the feedback information. The vector update strength is jointly determined by the number of feedbacks, the importance degree of the feedback, and the learning rate. Enhance and normalize the query vector according to the vector update strength to generate an updated query vector.

[0083] During the matching process between the query vector and the knowledge vector library, first, it is necessary to vectorize the input natural language query. Convert the query text into a high-dimensional vector representation through a pre-trained language model. The vector dimension is usually selected as 768 or 1024 dimensions. The knowledge vector library stores the vector representations of information such as standard descriptions of business indicators, associated data tables, and field definitions.

[0084] When performing vector matching, use a block calculation method to improve efficiency. Divide the knowledge vector library into blocks according to the business domain. Each block contains related business indicator information. Calculate the cosine similarity between the query vector and the vectors in each block to obtain preliminary matching results. Set the similarity threshold to 0.75, and only retain the matching items with similarity greater than the threshold.

[0085] When constructing a query graph based on the matching results, identify the data tables as the vertices of the graph and the inter-table association relationships as the edges. For example, if the order table and the user table are associated through the user ID, add an edge between the corresponding vertices. Obtain the connectivity index by calculating the ratio of the number of edges to the number of vertices. When the index is less than 0.8, it indicates that the query graph is not complete and association conditions need to be supplemented.

[0086] For the fields involved in the query, evaluate their effectiveness from multiple dimensions. The field correlation is calculated by the co-occurrence frequency with business metrics, the query frequency is based on the statistics of recent query logs, and the null value rate is calculated from sampled data. The weighted average of these three metrics is used as the effectiveness weight of the field.

[0087] The hierarchical interrogation strategy divides the supplementary conditions into three layers: the importance coefficient of the core business condition confirmation layer is 0.5, the associated relationship supplementary layer is 0.3, and the query constraint refinement layer is 0.2. In each layer, determine the interrogation priority based on the field effectiveness weight and the business impact degree. For example, for the order analysis scenario, first confirm the order time range, then supplement the associated conditions in the user dimension, and finally refine the constraint conditions such as the order status.

[0088] After obtaining user feedback, calculate the vector update intensity. The number of feedbacks represents the number of times the user confirms the same problem, the feedback importance degree is determined based on the user's selection operation, and the learning rate is initially set to 0.1 and decreases with the number of feedbacks. Adjust the corresponding dimension of the query vector according to the update intensity to ensure that the updated vector still satisfies the normalization constraint.

[0089] This application can achieve:

[0090] Through multi-dimensional vector matching and dynamic threshold adjustment, the accuracy of query intent understanding is improved, query errors caused by ambiguous understanding are reduced, and the system can more accurately identify the user's true query needs.

[0091] By adopting the hierarchical interrogation strategy and the priority sorting mechanism, the cost of user interaction is reduced, redundant confirmation processes are avoided, the efficiency of query condition supplementation is improved, and the integrity and rationality of the supplementary conditions are ensured at the same time.

[0092] Based on the vector dynamic update mechanism of user feedback, continuous optimization of query understanding ability is achieved. The system can adaptively adjust the query strategy, improve the accuracy of query results and user satisfaction, and reduce the workload of system maintenance at the same time.

[0093] In an optional implementation manner, based on the query vector and the matching result, extract the structure information and field definitions of the target data table from the knowledge vector library; construct an initial association path according to the data flow path in the data lineage information, including:

[0094] Decompose the query vector into an entity vector and a relationship vector, perform a linear combination of the entity vector and the relationship vector based on a preset weight coefficient to generate a target query vector; calculate the correlation score between the target query vector and each data table in the knowledge vector library, where the correlation score is obtained by weighted calculation of the entity matching score, the relationship matching score, and the field coverage rate, and filter out the data tables with a correlation score greater than the correlation threshold as candidate data tables;

[0095] Construct a structural similarity matrix for the candidate data tables, where each element value in the structural similarity matrix is obtained by calculating the ratio of the intersection to the union of the field sets of the corresponding two data tables; calculate the semantic feature vector of each field based on the structural similarity matrix, and perform a weighted sum of the semantic feature vectors to obtain a field semantic vector, and calculate the field semantic correlation degree between the candidate data tables according to the field semantic vector;

[0096] Construct a data lineage directed weighted graph based on the field semantic correlation degree, where the nodes of the directed weighted graph represent candidate data tables, the edges represent the inter-table association relationships, and the edge weight values are obtained by weighted calculation of the data dependence strength, the data transfer frequency, and the time correlation; calculate the path cost values of all association paths in the graph according to the edge weight values and the reliability coefficient calculated in advance;

[0097] Select the path with the minimum path cost value from the association paths as the candidate association path, calculate the quality score of the candidate association path, where the quality score is obtained by the ratio of the product of all edge weight values in the path to the sum of the squares of the connection overheads; at the same time, calculate the redundancy of the candidate association path, where the redundancy is determined based on the ratio of the necessary edge set required to maintain the data table association relationship to the actual edge set;

[0098] When the quality score is greater than the preset quality threshold and the redundancy is less than the preset redundancy threshold, determine the candidate association path as the final association path; otherwise, select the path with the second smallest path cost value from the remaining candidate association paths as the new candidate association path, and repeat the quality score calculation and redundancy calculation until the final association path that meets the quality and redundancy requirements is found.

[0099] First, introduce the specific implementation of query vector processing and target data table matching. Perform word segmentation and semantic parsing on the input query text to extract entity information and relationship information. The entity information includes specific objects such as table names and field names, and the relationship information includes operation types such as join and group by. Convert the entity information into a 300-dimensional entity vector, and convert the relationship information into a 100-dimensional relationship vector. In practice, the entity vector weight can be set to 0.7, and the relationship vector weight can be set to 0.3, and the final target query vector is generated through weighted combination.

[0100] Illustrated by a specific case, assume the query content is "Query the association between user order information and payment records". Through word segmentation, entity words such as "user", "order", "payment", etc. and relationship words such as "association" can be obtained. After converting these words into vectors and performing weighted combination, the target query vector is obtained.

[0101] Next, calculate the relevance between the target query vector and each data table in the knowledge base. The relevance calculation considers three aspects: entity matching degree, relationship matching degree, and field coverage rate. The entity matching degree reflects the similarity of table names and field names; the relationship matching degree reflects the matching degree of the association types between tables; the field coverage rate indicates the coverage of the fields required for the query in the target table. Set the weights of these three items to 0.4, 0.3, and 0.3 respectively. When the comprehensive score exceeds 0.75, include this table in the candidate set.

[0102] After obtaining the candidate data tables, it is necessary to analyze the structural similarity between the tables. By comparing the field sets of two tables, calculate the ratio of the number of common fields to the total number of fields as the structural similarity. For example, the user table and the order table may jointly contain the user_id field, and their total number of fields are 10 and 8 respectively, then their structural similarity is 0.1.

[0103] Based on the structural similarity matrix, further analyze the semantic association between fields. Extract information such as the name, type, and annotation of each field and convert it into a semantic feature vector. In practical applications, a pre-trained word vector model can be used to extract features and perform weighting according to the field importance. Finally, an association degree matrix reflecting the semantic association strength between fields is obtained.

[0104] When constructing a data lineage directed weighted graph, use the candidate data tables as nodes and the inter-table association relationships as edges. The weight of the edge needs to consider multiple factors: the data dependence strength (accounting for 0.4) reflects the reference relationship between tables; the data transfer frequency (accounting for 0.3) represents the data update frequency; the time correlation (accounting for 0.3) reflects the data timeliness. The edge weight is obtained through the weighted calculation of these factors.

[0105] When looking for the optimal association path, first calculate the cost values of each possible path. The path cost value considers the path length, edge weight, and reliability coefficient. The reliability coefficient is obtained based on the evaluation of historical data quality and reflects the data credibility. Select the path with the smallest cost value as the candidate association path.

[0106] Conduct a quality assessment on the candidate association path, calculate its quality score and redundancy. The quality score reflects the usability of the path and is measured by the balance between the cumulative effect of the edge weights in the path and the connection overhead. The redundancy represents the proportion of redundant connections in the path and is determined by comparing the scales of the necessary edge set and the actual edge set.

[0107] When the quality score exceeds 0.8 and the redundancy is lower than 0.2, the final associated path can be determined. Otherwise, the path with the second lowest cost needs to be selected for re-evaluation until a path that meets the conditions is found.

[0108] This application can achieve:

[0109] By means of vectorization processing and multi-dimensional matching, the accuracy of data table association is improved, and the query intention can be accurately identified and the most relevant data table can be found. By adopting the method of structural similarity and semantic association analysis, the in-depth mining of the relationship between data tables is realized, and the misjudgment caused by relying only on surface features is avoided. By introducing a quality evaluation and redundancy control mechanism, it is ensured that the generated associated path not only meets the business requirements but also has high execution efficiency, improving the query performance and resource utilization rate.

[0110] In an optional implementation manner, according to the index information in the metadata information, index matching is performed on the involved data screening conditions and table connection conditions, and the query method with the highest index coverage rate is selected as the index matching result; based on the table association order and the index matching result, generating a complete SQL query statement including data screening conditions, multi-table connection relationships and target fields includes:

[0111] Obtain the index information and query condition set of the data table, calculate the matching degree between each index and the query condition, calculate the index usage cost according to the storage overhead, access time and maintenance cost of the index, and select the index combination with the lowest index usage cost and an index coverage rate exceeding the preset coverage threshold as the candidate index set;

[0112] Based on the candidate index set, analyze the query conditions, calculate the selectivity score of the data screening conditions, and the selectivity score is determined according to the ratio of the amount of data that meets the screening conditions to the total amount of data in the table; evaluate the table connection conditions, and calculate the connection cost value based on the sizes of the tables participating in the connection, the connection index cardinality and the connection type, and filter out the connection condition combinations with a connection cost value lower than the preset cost threshold;

[0113] Use the connection condition combination to construct a multi-table connection tree, use the connection cost value as the weight of the connection edge, calculate the size of the intermediate result set of different connection sequences, and sort all connection sequences in combination with the selection rate and calculation cost of the connection step, and select the sequence with the lowest total cost as the optimal connection sequence;

[0114] Generate an initial SQL query statement according to the optimal connection sequence and the candidate index set, calculate the query integrity score of this query statement, and the integrity score includes three dimensions: query field coverage rate, connection condition integrity and index utilization rate; at the same time, construct a performance prediction model based on the theoretical execution time and IO overhead, and calculate the expected execution efficiency of the query statement;

[0115] When the query integrity score is lower than the preset score threshold or the expected execution efficiency is lower than the preset efficiency threshold, an adaptive tuning process is initiated by adjusting the connection sequence, updating the index selection, and optimizing the query structure until the performance requirements are met or the maximum number of tuning times is reached.

[0116] The present invention provides a SQL query generation method based on index matching and table join optimization. The method first obtains the metadata information of the data table, including index information and a set of query conditions. Then, the matching degree between each index and the query conditions is calculated, and the index usage cost is calculated considering the storage overhead, access time, and maintenance cost of the index.

[0117] During the index matching process, the system assigns a matching score to each index. For example, assume there is a user table (user_table) containing fields id, name, age, and city. The table has two indexes: idx_name_age (name, age) and idx_city (city). If the query condition is "WHERE name = 'John' AND age>30", the idx_name_age index will have a higher matching score because it fully covers the query condition.

[0118] Next, the system calculates the usage cost of each index. Here, the storage space, access speed, and maintenance difficulty of the index are considered. For example, for the idx_name_age index, if it occupies a large storage space (such as 1GB), but can significantly improve the query speed (such as reducing the query time from 5 seconds to 0.1 seconds), and the maintenance cost is moderate, then its usage cost may be evaluated as medium.

[0119] The system selects the index combination with the minimum index usage cost and an index coverage rate exceeding the preset coverage threshold as the candidate index set. Assume the preset coverage threshold is 80%. In the above example, the idx_name_age index may be selected as the candidate because it fully covers the query condition.

[0120] Based on the candidate index set, the system deeply analyzes the query conditions. First, the selectivity score of the data filtering condition is calculated. The selectivity score is determined by the ratio of the amount of data that meets the filtering condition to the total amount of data in the table. For example, if the user_table has a total of 1 million records, and 5000 records meet the condition "name = 'John' AND age>30", then the selectivity score of this filtering condition is 0.005 (5000 / 1000000).

[0121] For table join conditions, the system calculates the join cost based on the sizes of the tables involved in the join, the join index cardinality, and the join type. Suppose there is another order table (order_table) that needs to be joined with the user_table. If the user_table has 1 million records and the order_table has 10 million records, the join condition is user_table.id = order_table.user_id, and there is an index on user_id, then the cost of this join may be relatively low. The system filters out the combinations of join conditions whose costs are lower than the preset cost threshold.

[0122] Using the filtered combinations of join conditions, the system constructs a multi-table join tree. In this join tree, each join operation is represented as an edge, and the weight of the edge is the join cost calculated previously. The system calculates the sizes of the intermediate result sets for different join sequences and sorts all possible join sequences by combining the selectivity and computational cost of each join step. Finally, the system selects the sequence with the minimum total cost as the optimal join sequence.

[0123] For example, if there are three tables, user_table, order_table, and product_table, that need to be joined, the system may try the following join sequences:

[0124] 1. (user_table JOIN order_table) JOIN product_table;

[0125] 2. (order_table JOIN product_table) JOIN user_table;

[0126] 3. (user_table JOIN product_table) JOIN order_table;

[0127] Suppose that after calculation, the total cost of the first join sequence is the minimum, then it will be selected as the optimal join sequence.

[0128] Based on the optimal join sequence and the candidate index set, the system generates an initial SQL query statement. Then, the system calculates the query integrity score for this query statement. The integrity score includes three dimensions: query field coverage, join condition integrity, and index utilization.

[0129] The system checks whether this query covers all the required fields, whether it includes all the necessary join conditions, and whether it makes full use of the available indexes.

[0130] Meanwhile, the system constructs a performance prediction model based on the theoretical execution time and IO overhead to calculate the expected execution efficiency of the query statement. This prediction model may consider factors such as the hardware configuration of the database, the current load, and the historical query performance.

[0131] If the query integrity score is lower than the preset score threshold (e.g., 0.8) or the expected execution efficiency is lower than the preset efficiency threshold (e.g., the expected execution time exceeds 5 seconds), the system will initiate an adaptive tuning process. During this process, the system may attempt the following optimization strategies:

[0132] 1. Adjust the join sequence: For example, change the original (user_table JOIN order_table) JOIN product_table to (order_table JOIN product_table) JOIN user_table.

[0133] 2. Update the index selection: For example, if it is found that there is no index on the order_date field, it may suggest creating a new index to improve the query efficiency.

[0134] 3. Optimize the query structure: For example, rewrite a subquery as a JOIN, or split complex conditions into multiple simple conditions.

[0135] The system will continuously repeat this optimization process until the performance requirements are met or the maximum number of tuning times (e.g., 10 times) is reached. After each optimization, the system will re-evaluate the query integrity score and the expected execution efficiency to determine whether further optimization is needed.

[0136] Through this method, the system can generate SQL query statements that not only meet the query requirements but also have high execution efficiency.

[0137] This application can achieve:

[0138] By comprehensively analyzing the index information and query conditions, the optimal index combination can be selected, significantly improving the query efficiency. It not only considers the matching degree between the index and the query conditions, but also weighs the storage overhead, access time, and maintenance cost of the index, thus achieving a good balance between query performance and system resource utilization.

[0139] By constructing a multi-table join tree and evaluating different join sequences, the optimal table join order can be found. This method considers the size of the tables, the selectivity of the join conditions, and the size of the intermediate result sets, effectively reducing the amount of data processing during the query process, thus significantly improving the execution efficiency of complex queries.

[0140] An adaptive tuning mechanism is introduced, which can dynamically optimize query statements according to the query integrity score and the expected execution efficiency. This continuous optimization method can not only handle complex query scenarios, but also adapt to changes in the database environment, ensuring the long-term stability of query performance. In this way, this method can generate efficient SQL query statements in various complex database environments, greatly improving the overall performance of the database and the user experience.

[0141] In an alternative embodiment, a multi-table join tree is constructed using the connection condition combinations, the connection cost value is used as the weight of the connection edge, the sizes of the intermediate result sets of different connection sequences are calculated, and all possible connection sequences are sorted by combining the selectivity and computational cost of the connection steps, and the sequence with the minimum total cost is selected as the optimal connection sequence, including:

[0142] A connection correlation evaluation model is constructed based on the number of foreign key associations and data correlations between data tables, and the connection correlation score between any two data tables is calculated. The connection correlation score is jointly determined by the ratio of the number of foreign key associations to the number of table records and the data correlation coefficient;

[0143] The weight value of the connection edge is calculated according to the connection correlation score. The weight value is obtained by weighted calculation of the connection operation cost, input / output overhead, and processor calculation overhead, and the weight value is used as the weight attribute of the connection tree edge;

[0144] The cumulative execution cost is calculated for all connection sequences. The cumulative execution cost includes the connection operation cost, intermediate result transmission cost, and storage cost. The sequence execution efficiency score is calculated by combining the selectivity and processing cost of the connection steps;

[0145] The sequence with the highest execution efficiency score is selected from all connection sequences as the candidate sequence, and the expected resource consumption value of the candidate sequence is calculated. The expected resource consumption value includes the processor utilization rate, memory occupancy rate, and network bandwidth utilization rate;

[0146] When the expected resource consumption value exceeds the system capacity threshold, a sequence optimization process is triggered, and the resource consumption is reduced by adjusting the connection order, splitting the connection steps, or parallel processing until the optimal connection sequence that meets the system resource constraints is found.

[0147] Exemplarily, in the optimization of multi-table join queries in a database, it is first necessary to construct an evaluation model for the correlation of table joins. By analyzing the foreign key association situation between data tables, counting the number of foreign keys between each pair of tables, and calculating the ratio to the total number of records in the table. At the same time, combining the distribution characteristics of field values, the correlation coefficient between fields is calculated by means of data sampling. Taking the order table and the user table as an example, assuming that the order table has 10 million records, the user table has 1 million records, the two tables are associated through the user ID, and the correlation coefficient is 0.8, a relatively high join correlation score can be obtained.

[0148] After obtaining the join correlation score, the weight value of the join edge is further calculated. The weight calculation needs to consider multiple factors: the CPU overhead of the join operation itself, the IO overhead of data reading, and the storage overhead of intermediate results. Taking the join of two large tables as an example, if the sizes of the input tables are 10GB and 20GB respectively, and there is an index on the join field, the CPU overhead of the join operation is relatively small, and the main overhead comes from data reading, and a reasonable weight value can be calculated accordingly.

[0149] After constructing the join tree based on the above weight values, it is necessary to evaluate the execution efficiency of different join sequences. For each possible join sequence, the total execution cost is accumulated and calculated. Taking the join of three tables as an example, assuming that the sizes of tables A, B, and C are 1GB, 2GB, and 5GB respectively, the sizes of the intermediate result sets and the transmission costs of the sequences "A→B→C" and "B→C→A" will have significant differences. By simulating and calculating the selection rate of each join step (for example, 0.1 means that the result set is only 10% of the input data volume), the execution efficiency of different sequences can be accurately evaluated.

[0150] After obtaining the candidate sequence with the highest execution efficiency score, it is also necessary to estimate its resource consumption situation. By analyzing historical execution data, a resource consumption prediction model is established. For example, for a join operation with a total data volume of 50GB, it is expected to occupy 40% of the CPU resources, 30GB of memory space, and 2Gbps of network bandwidth. If the predicted value exceeds the system threshold (such as the memory usage rate shall not exceed 80%), the join sequence needs to be optimized.

[0151] The optimization methods include adjusting the join order, splitting a large join step into multiple small steps, or introducing a parallel processing mechanism. Taking the excessive memory occupation as an example, a large join operation can be split into multiple small batches for execution, and each batch processes part of the data, so as to control the memory peak within an acceptable range. Through iterative optimization, the optimal join sequence that meets both performance requirements and does not exceed resource limits is finally obtained.

[0152] This application can achieve:

[0153] By constructing an accurate connection relevance evaluation model and a comprehensive cost calculation system, the execution efficiency of multi-table join queries has been significantly improved, unnecessary resource waste has been reduced, and query performance has been greatly optimized. Based on resource consumption prediction and dynamic optimization mechanisms, the system can adaptively adjust connection strategies, effectively avoid resource overload problems, improve system stability and reliability, and ensure smooth operation in high-concurrency scenarios. By adopting a flexible connection sequence optimization scheme and combining technical means such as parallel processing and batch execution, efficient processing of large-scale data has been achieved, enhancing the scalability and adaptability of the system and meeting the needs of data processing of different scales.

[0154] Figure 3 This is a schematic structural diagram of the natural language SQL conversion system based on the data platform and large language model according to the embodiments of the present invention. As Figure 3 shown, the system includes:

[0155] The first unit is used to convert the obtained metadata information, data lineage information, business metric information, and historical query information into vector form through the data service API and store them in the knowledge vector library; use natural language processing technology to discriminate the intention of the query request input by the user, and convert the extracted intention information into a query vector; perform similarity matching between the query vector and the metadata vector, lineage vector, metric vector, and historical query vector in the knowledge vector library; determine the data tables, fields, and association relationships to be queried based on the matching results; when the matching results show that the query conditions are incomplete, generate supplementary inquiries according to the business metric information in the knowledge vector library; obtain the feedback information of the user on the supplementary inquiries, and update the query vector; repeat the similarity matching process until complete query conditions are obtained;

[0156] The second unit is used to extract the structural information and field definitions of the target data table from the knowledge vector library based on the query vector and the matching results; construct an initial association path according to the data flow path in the data lineage information; retrieve historical query records containing the same data table combination from the historical query information based on the data tables in the initial association path; sort the historical query records according to the query execution time, and select the table connection order in the historical query record with the shortest execution time as the optimized table association order;

[0157] A third unit is configured to perform index matching on the data filtering conditions and table join conditions involved according to the index information in the metadata information, and select the query method with the highest index coverage rate as the index matching result; generate a complete SQL query statement including data filtering conditions, multi-table join relationships, and target fields based on the table association order and the index matching result; execute the SQL query statement and record the query result; update the SQL query statement, the query vector, the query execution time, the association path, and the query result to the historical query information.

[0158] In a third aspect of the embodiments of the present invention,

[0159] a kind of electronic device is provided, including:

[0160] a processor;

[0161] a memory for storing instructions executable by the processor;

[0162] wherein, the processor is configured to call the instructions stored in the memory to execute the method described above.

[0163] In a fourth aspect of the embodiments of the present invention,

[0164] a computer-readable storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described above is implemented.

[0165] The present invention may be a method, an apparatus, a system, and / or a computer program product. The computer program product may include a computer-readable storage medium, on which computer-readable program instructions for executing various aspects of the present invention are uploaded.

[0166] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A natural language SQL conversion method based on a data platform and a large language model, characterized in that: include: The acquired metadata information, data lineage information, business indicator information and historical query information are converted into vector form through the data service API and stored in the knowledge vector library; Using natural language processing technology to identify the intent of the query request input by the user, converting the extracted intent information into a query vector; performing similarity matching on the query vector with the metadata vector, lineage vector, indicator vector and historical query vector in the knowledge vector library; determining the data table, field and association relationship to be queried based on the matching result; when the matching result shows that the query condition is incomplete, generating a supplementary query based on the business indicator information in the knowledge vector library; obtaining user feedback information on the supplementary query, and updating the query vector; repeating the similarity matching process until the complete query condition is obtained; Based on the query vector and the matching result, the structural information and field definition of the target data table are extracted from the knowledge vector library; an initial association path is constructed according to the data flow path in the data lineage information; based on the data table in the initial association path, historical query records containing the same data table combination are retrieved from the historical query information; the historical query records are sorted according to the query execution time, and the table connection sequence in the historical query record with the shortest execution time is selected as the optimized table association sequence; According to the index information in the metadata information, index matching is performed on the data screening conditions and table connection conditions involved, and the query method with the highest index coverage rate is selected as the index matching result; Based on the table association order and the index matching result, a complete SQL query statement including data screening conditions, multi-table connection relationships and target fields is generated; the SQL query statement is executed and the query result is recorded; Updating the SQL query statement, the query vector, the query execution time, the associated path and the query result to the historical query information; Based on the query vector and the matching result, extracting the structure information and field definition of the target data table from the knowledge vector library; Constructing an initial association path according to the data flow path in the data lineage information includes: Decomposing the query vector into an entity vector and a relationship vector, and performing a linear combination of the entity vector and the relationship vector based on a preset weight coefficient to generate a target query vector; calculating the relevance score between the target query vector and each data table in the knowledge vector library, wherein the relevance score is obtained by weighted calculation of the entity matching score, the relationship matching score, and the field coverage, and screening the data table whose relevance score is greater than the relevance threshold as a candidate data table; Constructing a structural similarity matrix for the candidate data tables, wherein each element value in the structural similarity matrix is ​​obtained by calculating the ratio of the intersection and the union of the field sets of the corresponding two data tables; calculating a semantic feature vector for each field based on the structural similarity matrix, performing weighted summation on the semantic feature vectors to obtain a field semantic vector, and calculating the field semantic association between the candidate data tables based on the field semantic vectors; A data lineage directed weighted graph is constructed based on the semantic association of the fields, wherein the nodes of the directed weighted graph represent candidate data tables, the edges represent associations between tables, and the edge weights are obtained by weighted calculation of data dependency strength, data flow frequency, and time correlation; and the path cost values ​​of all association paths in the graph are calculated based on the edge weights and the pre-calculated reliability coefficients; Selecting a path with the smallest path cost from the association paths as a candidate association path, calculating a quality score of the candidate association path, wherein the quality score is obtained by the ratio of the cumulative multiplication of all edge weight values ​​in the path to the sum of squares of connection costs; and calculating the redundancy of the candidate association path, wherein the redundancy is determined based on the ratio of a required edge set to maintain the association relationship of the data table to an actual edge set; When the quality score is greater than a preset quality threshold and the redundancy is less than a preset redundancy threshold, the candidate association path is determined as the final association path; otherwise, a path with the second smallest path cost is selected from the remaining candidate association paths as a new candidate association path, and the quality score calculation and redundancy calculation are repeatedly performed until a final association path that meets the quality and redundancy requirements is found.

2. The method according to claim 1, characterized in that The metadata information includes data table structure, field definition, index information, constraints, data types and field comments; The data lineage information includes data flow paths, upstream and downstream associations of data objects, and task dependencies; The business indicator information includes sales statistics, conversion rate calculation and inventory turnover rate; The historical query information includes SQL query statements, query frequency and query execution time.

3. The method according to claim 1, characterized in that Determine the data table, field and associated relationship to be queried based on the matching results; when the matching results show that the query conditions are incomplete, generate a supplementary query based on the business indicator information in the knowledge vector library; Acquiring user feedback information on the supplementary query and updating the query vector includes: The query vector is matched with the vectors in the knowledge vector library by using a similarity calculation method. The query vector and the vectors in the knowledge vector library are matched in multiple dimensions by cosine similarity calculation, and the matching result is determined according to a set similarity threshold; Based on the matching results, a query graph is constructed, wherein the query graph includes a vertex set representing a data table and an edge set representing an association relationship between tables, and a connectivity index of the query graph is calculated, wherein the connectivity index represents the integrity of the graph through the relationship between the number of edges and the number of vertices; when the connectivity index is less than a preset connectivity threshold, a data table node that needs to supplement the association condition is marked; Verify the validity of the fields involved in the query, and calculate the field validity weight based on the field relevance, query frequency and null value rate. The field relevance represents the degree of association between the field and the business indicator, the query frequency represents the frequency of use of the field in historical queries, and the null value rate represents the data quality of the field; Based on the field validity weight, a hierarchical query strategy is constructed, wherein the hierarchical query strategy includes a core business condition confirmation layer, an association relationship supplement layer, and a query constraint refinement layer, and each layer is assigned a different importance coefficient; the query priority is calculated according to the influence of the business indicator on the supplementary condition, and the corresponding supplementary query is generated according to the query priority; Obtain user feedback information on the supplementary query, calculate vector update strength based on the feedback information, where the vector update strength is determined by the number of feedbacks, the feedback importance, and the learning rate; enhance and normalize the query vector according to the vector update strength to generate an updated query vector.

4. The method according to claim 1, characterized in that According to the index information in the metadata information, index matching is performed on the data screening conditions and table connection conditions involved, and the query method with the highest index coverage rate is selected as the index matching result; Based on the table association order and the index matching result, a complete SQL query statement including data screening conditions, multi-table connection relationships and target fields is generated, including: Obtain the index information and query condition set of the data table, calculate the matching degree between each index and the query condition, calculate the index usage cost according to the index storage overhead, access time and maintenance cost, and select the index combination with the minimum index usage cost and index coverage exceeding the preset coverage threshold as the candidate index set; Analyze the query conditions based on the candidate index set, calculate the selectivity score of the data screening conditions, and determine the selectivity score according to the ratio of the amount of data that meets the screening conditions to the total amount of data in the table; evaluate the table connection conditions, calculate the connection cost value based on the table size involved in the connection, the connection index cardinality and the connection type, and select the connection condition combination whose connection cost value is lower than the preset cost threshold; Using the connection condition combination to construct a multi-table connection tree, using the connection cost value as the weight of the connection edge, calculating the size of the intermediate result set of different connection sequences, sorting all connection sequences based on the selectivity and calculation cost of the connection steps, and selecting the sequence with the smallest total cost as the optimal connection sequence; Generate an initial SQL query statement according to the optimal connection sequence and the candidate index set, and calculate the query integrity score of the query statement, wherein the integrity score includes three dimensions: query field coverage, connection condition integrity, and index utilization; at the same time, build a performance prediction model based on theoretical execution time and IO overhead to calculate the expected execution efficiency of the query statement; When the query integrity score is lower than a preset score threshold or the expected execution efficiency is lower than a preset efficiency threshold, the adaptive tuning process is started by adjusting the connection sequence, updating the index selection, and optimizing the query structure until the performance requirements are met or the maximum number of tuning times is reached.

5. The method according to claim 1, characterized in that Using the connection condition combination to construct a multi-table connection tree, using the connection cost value as the weight of the connection edge, calculating the size of the intermediate result set of different connection sequences, sorting all possible connection sequences based on the selectivity and calculation cost of the connection steps, and selecting the sequence with the smallest total cost as the optimal connection sequence includes: A connection relevance evaluation model is constructed based on the number of foreign key associations and data relevance between data tables to calculate the connection relevance score between any two data tables. The connection relevance score is determined by the ratio of the number of foreign key associations to the number of table records and the data relevance coefficient. Calculate the weight value of the connection edge according to the connection relevance score, the weight value is obtained by weighted calculation of the connection operation cost, input and output overhead and processor calculation overhead, and use the weight value as the weight attribute of the connection tree edge; Calculate the cumulative execution cost for all connection sequences, the cumulative execution cost includes the connection operation cost, the intermediate result transmission cost and the storage cost, and calculate the sequence execution efficiency score by combining the selection rate and processing cost of the connection steps; Selecting a sequence with the highest execution efficiency score from all connection sequences as a candidate sequence, and calculating an expected resource consumption value of the candidate sequence, wherein the expected resource consumption value includes a processor usage rate, a memory occupancy rate, and a network bandwidth usage rate; When the expected value of resource consumption exceeds the system capacity threshold, the sequence optimization process is triggered to reduce resource consumption by adjusting the connection order, splitting the connection steps or parallel processing until the optimal connection sequence that meets the system resource constraints is found.

6. A natural language SQL conversion system based on a data platform and a large language model, used to implement the method as claimed in any one of claims 1 to 5, characterized in that: include: The first unit is used to convert the acquired metadata information, data lineage information, business indicator information and historical query information into vector form through the data service API and store it in the knowledge vector library; Using natural language processing technology to identify the intent of the query request input by the user, converting the extracted intent information into a query vector; performing similarity matching on the query vector with the metadata vector, lineage vector, indicator vector and historical query vector in the knowledge vector library; determining the data table, field and association relationship to be queried based on the matching result; when the matching result shows that the query condition is incomplete, generating a supplementary query based on the business indicator information in the knowledge vector library; obtaining user feedback information on the supplementary query, and updating the query vector; repeating the similarity matching process until the complete query condition is obtained; The second unit is used to extract the structural information and field definition of the target data table from the knowledge vector library based on the query vector and the matching result; construct an initial association path according to the data flow path in the data lineage information; retrieve historical query records containing the same data table combination from the historical query information based on the data table in the initial association path; sort the historical query records according to the query execution time, and select the table connection order in the historical query record with the shortest execution time as the optimized table association order; The third unit is used to perform index matching on the data screening conditions and table connection conditions involved according to the index information in the metadata information, and select the query method with the highest index coverage as the index matching result; Based on the table association order and the index matching result, a complete SQL query statement including data screening conditions, multi-table connection relationships and target fields is generated; the SQL query statement is executed and the query result is recorded; The SQL query statement, the query vector, the query execution time, the associated path and the query result are updated to the historical query information.

7. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images