Financial data warehouse intelligent SQL optimization method and system based on large model

By applying large models to the financial warehouse for SQL optimization, the efficiency problems of traditional SQL optimization methods under the needs of high-dimensional data processing and complex query are solved, and more efficient SQL query execution and system performance improvement are achieved.

CN120045586APending Publication Date: 2025-05-27SHANGHAI WEIXINHUIZHI FINANCIAL TECH CO LTD
View PDF 0 Cites 10 Cited by

Patent Information

Application Number
CN202411921514.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-12-25
Publication Date
2025-05-27

AI Technical Summary

Technical Problem

Traditional SQL optimization methods seem to be ineffective when facing high-dimensional data processing, complex relationships of multiple tables, and frequently changing query needs, limiting the effective application of big data processing technology and artificial intelligence technology, affecting the efficiency of query execution and the rational use of system resources.

Method used

The intelligent SQL optimization method of financial data warehouses based on large models is adopted, and the execution efficiency of SQL query and the overall performance of the system is significantly improved through SQL statement analysis and preprocessing, large model semantic understanding and optimization suggestions, dynamic execution plan generation and selection, as well as intelligent resource allocation and parallel execution.

Benefits of technology

It significantly improves the execution efficiency of SQL query, reduces query response time, improves resource utilization, and enhances the adaptability and scalability of the system, adapting to the challenges of data growth in the financial field and complex query demand.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120045586A_ABST
    Figure CN120045586A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of data optimization, and discloses a financial data warehouse intelligent SQL (Structured Query Language) optimization method based on a large model, which comprises the following steps: S1, SQL statement analysis and preprocessing: carrying out word segmentation, grammatical analysis and logic structure analysis on an SQL statement submitted by a user, and identifying keywords, table structures and query conditions in query; s2, large model semantic understanding and optimization suggestion: performing semantic understanding and logical reasoning on the analyzed SQL statements by using a large model, and automatically generating optimization suggestions, the optimization suggestions including index recommendation, query rewriting and execution plan optimization; and S3, generating and selecting a dynamic execution plan. According to the method, the advantages of a large model in the aspects of natural language processing and intelligent decision making are fully utilized and combined with the specific requirements of the financial data warehouse, and the execution efficiency of SQL query and the overall performance of the system are remarkably improved; the query response time of the financial data warehouse can be effectively shortened, and the resource utilization rate is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data optimization, and particularly to an intelligent SQL optimization method and system for financial data warehouses based on large models. Background Art

[0002] With the rapid increase in the amount of data in the financial field, traditional SQL optimization means are unable to cope with challenges such as high-dimensional data processing, complex multi-table associations, and frequently changing query requirements. Currently, the practice of SQL optimization mainly relies on established rules and human experience, which limits the effective application of big data processing technology and artificial intelligence technology, and thus affects the query execution efficiency and the rational utilization of system resources.

[0003] To solve the above problems, an intelligent SQL optimization method and system for financial data warehouses based on large models are proposed in this application. Summary of the Invention

[0004] (I) Object of the Invention

[0005] To solve the technical problems in the background art, the present invention proposes an intelligent SQL optimization method and system for financial data warehouses based on large models. The present invention makes full use of the advantages of large models in natural language processing and intelligent decision-making, combines them with the specific requirements of financial data warehouses, significantly improves the execution efficiency of SQL queries and the overall performance of the system; can effectively reduce the query response time of financial data warehouses and improve resource utilization.

[0006] (II) Technical Solution

[0007] To solve the above problems, the present invention provides an intelligent SQL optimization method for financial data warehouses based on large models, including the following steps:

[0008] S1. SQL statement parsing and preprocessing:

[0009] Perform word segmentation, syntax analysis, and logical structure parsing on the SQL statement submitted by the user to identify the keywords, table structures, and query conditions in the query;

[0010] S2. Semantic understanding of the large model and optimization suggestions:

[0011] Use the large model to perform semantic understanding and logical reasoning on the parsed SQL statement, and automatically generate optimization suggestions. The optimization suggestions include index recommendations, query rewriting, and execution plan optimization;

[0012] S3. Dynamic execution plan generation and selection;

[0013] Based on the multiple optimized execution plans generated in S2, combined with historical query performance data, select the optimal execution plan and monitor its execution efficiency in real time;

[0014] S4. Energy Resource Allocation and Parallel Execution:

[0015] Based on the optimized SQL query, calculate resources are reasonably allocated, and parallel processing is adopted to improve query performance.

[0016] Preferably, in S1, it specifically includes:

[0017] First, perform a comprehensive word segmentation process on the SQL statement submitted by the user, accurately splitting the statement into individual word units for subsequent analysis;

[0018] Conduct in-depth syntax analysis, and based on SQL syntax rules, carefully check the structural legality of the statement to determine the relationships of various syntax components;

[0019] Perform logical structure parsing to identify keywords, table structures, and query conditions in the query;

[0020] Construct a syntax tree to clearly represent the syntax and logical relationships of the SQL statement in a hierarchical structure.

[0021] Preferably, in S2, it specifically includes:

[0022] Utilize a large model to conduct in-depth semantic understanding of the parsed SQL statement;

[0023] Apply deep learning algorithms to further mine the semantics of the SQL statement, analyze the semantic associations and importance weights of different parts of the statement, and generate optimization suggestions;

[0024] The optimization suggestions include index recommendations: Based on the table structure of the query and the frequently queried fields, recommend appropriate index types and index column combinations;

[0025] Query rewriting: Perform equivalent transformation on the original query statement;

[0026] Execution plan optimization: Propose optimization solutions for different query execution steps and data access paths.

[0027] Preferably, in S3, it specifically includes:

[0028] Based on multiple optimized execution plans generated by the large model, conduct comprehensive evaluation in combination with rich historical query performance data, and the historical query performance data includes detailed information such as response time, resource consumption, and data read volume when different queries were executed in the past;

[0029] Apply a performance prediction model to predict the performance of each execution plan in the current data state and system environment;

[0030] The performance prediction model is constructed based on machine learning algorithms. According to the relationship between features and performance metrics in historical data, it predicts key performance parameters such as the execution time and resource requirements of each execution plan.

[0031] Select the execution plan with the optimal predicted performance, and monitor its execution efficiency in real time during execution. Through the system monitoring tool, collect data on the actual execution time and resource usage of the execution plan in real time, and compare and analyze it with the predicted value. If it is found that the actual execution efficiency deviates significantly from the predicted value, adjust or reselect the execution plan in a timely manner.

[0032] Preferably, in S4, it specifically includes:

[0033] Dynamically adjust the allocation of computing resources according to the complexity of the query. For simple queries, reduce the occupancy of computing resources; for complex queries, intelligently increase resources such as the number of CPU cores and memory allocation to ensure that the query can be executed efficiently.

[0034] By real-time evaluating factors such as the number of tables, the complexity of join conditions, and the amount of data in the query statement, use a resource allocation algorithm to determine an appropriate resource allocation plan.

[0035] Select an appropriate parallel processing method according to the parallelism of the query. For queries with high parallelism, a parallel processing method based on data partitioning can be adopted to distribute the query tasks to multiple computing nodes or threads for simultaneous execution.

[0036] For queries with data dependency relationships, adopt pipeline parallel processing and arrange the parallel execution order of each computing step to maximize the utilization of computing resources.

[0037] A financial data warehouse intelligent SQL optimization system based on large models, including:

[0038] An SQL parsing and preprocessing module, which is used to perform word segmentation, syntax analysis, and logical structure parsing on the SQL statements submitted by users, and identify keywords, table structures, and query conditions in the query.

[0039] A large model semantic understanding and optimization suggestion module, which is used to utilize the large model to perform semantic understanding and logical reasoning on the parsed SQL statements and automatically generate optimization suggestions.

[0040] A dynamic execution plan generation and selection module, which is used to select the optimal execution plan based on multiple generated optimized execution plans and historical query performance data, and monitor its execution efficiency in real time.

[0041] An intelligent resource allocation and parallel execution module, which is used to reasonably allocate computing resources on the basis of the optimized SQL query and adopt a parallel processing method to improve query performance.

[0042] Preferably, the SQL parsing and preprocessing module includes:

[0043] A word segmentation unit for splitting the SQL statement submitted by the user into individual words or phrases;

[0044] A syntax analysis unit for parsing the syntax structure of the SQL statement and generating a syntax tree;

[0045] A logical structure analysis unit for analyzing the logical structure of the SQL statement;

[0046] A syntax tree construction unit for constructing the parsed SQL statement into a syntax tree.

[0047] Preferably, the large model semantic understanding and optimization suggestion module includes:

[0048] A semantic understanding unit for using the large model to perform semantic understanding on the SQL statement and identify its intention and meaning;

[0049] A deep learning algorithm unit for encoding and parsing the SQL statement using deep learning algorithms;

[0050] An optimization suggestion generation unit: for generating index recommendations, query rewriting, and execution plan optimization suggestions according to the semantic representation, combined with optimization rules and strategies.

[0051] Preferably, the dynamic execution plan generation and selection module includes:

[0052] An execution plan generation unit for generating multiple optimized execution plans based on the SQL statement and the optimization suggestions of the large model;

[0053] A historical query performance database for storing historical query performance data, including query statements, execution plans, and execution times;

[0054] A performance prediction unit for performing performance prediction on the execution plan and evaluating its execution efficiency and resource consumption;

[0055] An execution plan selection unit for selecting the optimal execution plan by combining historical query performance data and performance prediction results;

[0056] A real-time monitoring unit for real-time monitoring of the execution efficiency of the execution plan.

[0057] Preferably, the intelligent resource allocation and parallel execution module includes:

[0058] A resource evaluation unit for calculating its complexity according to the syntax and logical structure of the query statement and evaluating the required computing resources;

[0059] A resource allocation unit for reasonably allocating computing resources according to the complexity of queries and the system resource usage situation;

[0060] A parallel processing unit for selecting appropriate parallel processing according to the parallelism of queries;

[0061] A resource monitoring and adjustment unit for monitoring the resource usage situation in real time and making dynamic adjustments according to actual requirements.

[0062] The above technical solution of the present invention has the following beneficial technical effects:

[0063] Significantly improve the SQL query execution efficiency:

[0064] By leveraging the advantages of large models in natural language processing and intelligent decision-making, this method can deeply understand the semantics and intentions of SQL queries, thereby generating more accurate optimization suggestions.

[0065] These optimization suggestions include index recommendations, query rewriting, and execution plan optimization, which can significantly reduce the query response time and improve the query speed.

[0066] Improve the overall system performance:

[0067] This method not only optimizes the performance of individual SQL queries, but also improves the performance of the entire financial data warehouse system through dynamic execution plan generation and selection, as well as intelligent resource allocation and parallel execution strategies.

[0068] The system can automatically adjust resource allocation according to the actual situation to ensure the efficient execution of complex queries and avoid resource waste at the same time.

[0069] Reduce the query response time:

[0070] Through accurate SQL parsing and preprocessing, as well as the semantic understanding of large models, the system can quickly identify key information and optimization points in queries.

[0071] This enables the system to generate and execute optimized query plans in a shorter time, thereby reducing the query response time and improving the user experience.

[0072] Improve resource utilization:

[0073] This method dynamically adjusts resource allocation according to the complexity of queries and the system resource usage situation through an intelligent resource allocation and parallel execution module.

[0074] This not only ensures the efficient execution of queries, but also avoids over-occupation and waste of resources, improving resource utilization and the cost performance of the system.

[0075] Enhance the adaptability and scalability of the system:

[0076] This method is constructed based on large models and machine learning algorithms and can automatically adjust optimization strategies as the data volume increases and query requirements change.

[0077] This enables the system to have stronger adaptability and scalability, and be able to better cope with the continuous growth of data volume and the complexity of query requirements in the future financial field.

[0078] In summary, it can significantly improve the SQL query execution efficiency, enhance the overall system performance, reduce the query response time, increase the resource utilization rate, and strengthen the system's adaptability and scalability. This is of great significance for data processing and query optimization in the financial field and helps to promote the digital transformation and intelligent upgrading of the financial industry. Brief Description of the Drawings

[0079] Figure 1 It is a flowchart of an intelligent SQL optimization method for a financial data warehouse based on large models proposed by the present invention. Detailed Embodiments

[0080] To make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are exemplary and not intended to limit the scope of the present invention. In addition, in the following descriptions, the descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present invention.

[0081] As Figure 1 shown, an intelligent SQL optimization method for a financial data warehouse based on large models proposed by the present invention is characterized by including the following steps:

[0082] S1. SQL statement parsing and preprocessing:

[0083] Perform word segmentation, syntax analysis, and logical structure parsing on the SQL statement submitted by the user to identify the keywords, table structures, and query conditions in the query;

[0084] Specifically, it includes:

[0085] First, perform a comprehensive word segmentation process on the SQL statement submitted by the user, accurately splitting the statement into individual word units for subsequent analysis;

[0086] Conduct in-depth syntax analysis, and based on SQL syntax rules, carefully check the structural legality of the statement and determine the relationships of each syntax component;

[0087] Perform logical structure parsing to identify the keywords (such as SELECT, FROM, WHERE) in the query, table structures (including table names, column names, data types), and query conditions (various logical expressions and comparison operators);

[0088] Construct a syntax tree to clearly represent the syntax and logical relationships of SQL statements in a hierarchical structure, providing a convenient structured data foundation for the subsequent semantic understanding and logical reasoning of large models. For example, construct a complex multi-table join query statement into a syntax tree, where nodes represent keywords, table names, or expressions, and edges represent the logical connection relationships between them.

[0089] S2. Semantic understanding of large models and optimization suggestions:

[0090] Use large models to perform semantic understanding and logical reasoning on the parsed SQL statements, and automatically generate optimization suggestions, which include index recommendations, query rewriting, and execution plan optimization.

[0091] In S2, it specifically includes:

[0092] Leverage large models to perform in-depth semantic understanding on the parsed SQL statements. Large models can accurately grasp the query intent and semantic connotations based on their vast knowledge systems and advanced neural network architectures.

[0093] Apply deep learning algorithms to further mine the semantics of SQL statements. For example, through recurrent neural networks (RNNs) or attention mechanisms, analyze the semantic associations and importance weights of different parts of the statement to generate more accurate optimization suggestions.

[0094] The optimization suggestions cover index recommendations: Based on the table structure of the query and the frequently queried fields, recommend appropriate index types (such as B-tree indexes, hash indexes) and index column combinations to accelerate data retrieval.

[0095] Query rewriting: Perform equivalent transformations on the original query statement. For example, convert complex subqueries into join queries to improve query execution efficiency.

[0096] Execution plan optimization: Propose optimization schemes for different query execution steps and data access paths, such as adjusting the table join order and selecting appropriate join algorithms (such as nested loop joins, hash joins).

[0097] S3. Generation and selection of dynamic execution plans;

[0098] Based on the multiple optimized execution plans generated in S2, combined with historical query performance data, select the optimal execution plan and monitor its execution efficiency in real time.

[0099] Specifically includes:

[0100] Multiple optimized execution plans generated based on large models are comprehensively evaluated in combination with rich historical query performance data. The historical query performance data includes details such as the response time, resource consumption (such as CPU usage, memory occupancy), and data read volume when different queries were executed in the past.

[0101] Use a performance prediction model to predict the performance of each execution plan under the current data state and system environment.

[0102] The performance prediction model can be constructed based on machine learning algorithms, such as using linear regression or decision tree algorithms, to predict key performance parameters such as the execution time and resource requirements of each execution plan according to the relationship between features (such as data volume, index situation, query complexity) in historical data and performance metrics.

[0103] Select the execution plan with the optimal predicted performance and monitor its execution efficiency in real time during execution. Through system monitoring tools, collect data on the actual execution time and resource usage of the execution plan in real time, and compare and analyze it with the predicted values. If it is found that the actual execution efficiency deviates significantly from the predicted value, make adjustments or reselect the execution plan in a timely manner. For example, if it is found that the CPU usage is too high during the execution of a certain execution plan, resulting in a slow system response, the execution plan switching mechanism can be triggered based on the real-time monitoring data, and a standby execution plan with lower CPU resource requirements can be selected to continue the execution.

[0104] S4. Energy resource allocation and parallel execution:

[0105] Based on the optimized SQL query, reasonably allocate computing resources and adopt parallel processing methods to improve query performance.

[0106] Specifically include:

[0107] Dynamically adjust the allocation of computing resources according to the complexity of the query. For simple queries, reasonably reduce the occupancy of computing resources to avoid resource waste; for complex queries, such as multi-table joins and aggregations involving large-scale data, intelligently increase resources such as the number of CPU cores and memory allocation to ensure that the query can be executed efficiently. For example, by real-time evaluating factors such as the number of tables, the complexity of join conditions, and the data volume in the query statement, use resource allocation algorithms (such as rule-based algorithms or machine learning-based dynamic allocation algorithms) to determine an appropriate resource allocation plan.

[0108] Select an appropriate parallel processing method according to the parallelism of the query. For queries with high parallelism, such as query operations on data in multiple independent partitions, a parallel processing method based on data partitioning can be adopted to distribute the query tasks to multiple computing nodes or threads for simultaneous execution; for queries with data dependencies, a pipeline parallel processing method is adopted to reasonably arrange the parallel execution order of each computing step to maximize the utilization of computing resources. For example, when querying a database that stores financial transaction data partitioned by time, if the query involves data in multiple time periods, the data query tasks for each time period can be distributed to different threads for parallel execution, and then the results are merged.

[0109] An intelligent SQL optimization system for financial data warehouses based on large models, including:

[0110] An SQL parsing and preprocessing module for tokenizing, syntax analyzing, and logically structuring the SQL statements submitted by users to identify keywords, table structures, and query conditions in the queries;

[0111] The SQL parsing and preprocessing module includes:

[0112] A tokenization unit for splitting the SQL statements submitted by users into individual words or phrases;

[0113] A syntax analysis unit for parsing the syntax structure of SQL statements to generate a syntax tree;

[0114] A logical structure parsing unit for analyzing the logical structure of SQL statements, such as the SELECT, FROM, and WHERE clauses;

[0115] A syntax tree construction unit for constructing the parsed SQL statements into a syntax tree to facilitate subsequent semantic understanding and logical reasoning.

[0116] A large model semantic understanding and optimization recommendation module for using a large model to perform semantic understanding and logical reasoning on the parsed SQL statements and automatically generating optimization recommendations;

[0117] The large model semantic understanding and optimization recommendation module includes:

[0118] A semantic understanding unit for using a large model to perform semantic understanding on SQL statements to identify their intentions and meanings;

[0119] A deep learning algorithm unit for encoding and parsing SQL statements using deep learning algorithms (such as Transformer, BERT);

[0120] An optimization recommendation generation unit: for generating index recommendations, query rewriting, and execution plan optimization recommendations based on semantic representations, combined with optimization rules and strategies.

[0121] The dynamic execution plan generation and selection module is used to select the optimal execution plan based on multiple generated optimized execution plans, combined with historical query performance data, and monitor its execution efficiency in real time;

[0122] The dynamic execution plan generation and selection module includes:

[0123] The execution plan generation unit generates multiple optimized execution plans based on SQL statements and optimization suggestions from the large model.

[0124] The historical query performance database is used to store historical query performance data, including query statements, execution plans, and execution times;

[0125] Use a database management system (such as MySQL, PostgreSQL) for storage and management.

[0126] The performance prediction unit is used to predict the performance of the execution plan, evaluate its execution efficiency and resource consumption.

[0127] Use machine learning algorithms (such as regression, classification) to evaluate the performance of the execution plan.

[0128] The execution plan selection unit is used to select the optimal execution plan by combining historical query performance data and performance prediction results.

[0129] The real-time monitoring unit is used to monitor the execution efficiency of the execution plan in real time to ensure the stability and reliability of query performance.

[0130] The intelligent resource allocation and parallel execution module is used to reasonably allocate computing resources based on the optimized SQL query and adopt parallel processing methods to improve query performance. Details of the method steps are given below.

[0131] The intelligent resource allocation and parallel execution module includes:

[0132] The resource evaluation unit calculates its complexity based on the syntax and logical structure of the query statement and evaluates the required computing resources;

[0133] The resource allocation unit reasonably allocates computing resources according to the complexity of the query and the system resource usage situation.

[0134] Use a resource manager (such as Kubernetes, Docker) for dynamic resource allocation and scheduling.

[0135] The parallel processing unit selects an appropriate parallel processing method according to the parallelism of the query to improve query performance.

[0136] Perform parallel processing using a parallel computing framework (such as Spark, Flink) and optimize the execution efficiency of queries.

[0137] A resource monitoring and adjustment unit for monitoring the usage of resources in real time and making dynamic adjustments according to actual requirements.

[0138] Use resource monitoring tools (such as Prometheus, Grafana) for real-time monitoring and dynamic adjustment of resources.

[0139] It should be understood that the above specific embodiments of the present invention are only for illustrative explanation or interpretation of the principles of the present invention, and do not constitute a limitation to the present invention. Therefore, any modifications, equivalent replacements, improvements, etc. made without departing from the spirit and scope of the present invention shall be included within the protection scope of the present invention. In addition, the appended claims of the present invention are intended to cover all variations and modifications that fall within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A financial data warehouse intelligent SQL optimization method based on a large model, characterized in that: The following steps are involved: S1. SQL statement parsing and preprocessing: Perform word segmentation, syntax analysis and logical structure analysis on SQL statements submitted by users, and identify keywords, table structures and query conditions in the query; S2. Semantic understanding and optimization suggestions for large models: Use the big model to perform semantic understanding and logical reasoning on the parsed SQL statements and automatically generate optimization suggestions, including index recommendations, query rewriting, and execution plan optimization; S3, dynamic execution plan generation and selection; Based on the various optimized execution plans generated by S2 and combined with historical query performance data, the optimal execution plan is selected and its execution efficiency is monitored in real time; S4, energy resource allocation and parallel execution: Based on the optimized SQL query, computing resources are reasonably allocated and parallel processing is adopted to improve query performance.

2. According to claim 1, a financial data warehouse intelligent SQL optimization method based on a large model is characterized in that: S1 specifically includes: First, the SQL statements submitted by users are fully segmented to accurately split the statements into individual word units for subsequent analysis; Conduct in-depth syntax analysis, check the structure of the statement in detail according to SQL syntax rules, and determine the relationship between each syntax component; Perform logical structure analysis to identify keywords, table structures, and query conditions in queries; Build a syntax tree to clearly represent the syntax and logical relationships of SQL statements in a hierarchical structure.

3. According to claim 2, a financial data warehouse intelligent SQL optimization method based on a large model is characterized in that: In S2, it specifically includes: Use the big model to deeply understand the semantics of the parsed SQL statements; Use deep learning algorithms to further mine the semantics of SQL statements, analyze the semantic associations and importance weights of different parts of the statements, and generate optimization suggestions; Optimization suggestions include index recommendations: recommending appropriate index types and index column combinations based on the query table structure and frequently queried fields; Query rewriting: perform equivalent transformation on the original query statement; Execution plan optimization: Propose optimization solutions for different query execution steps and data access paths.

4. According to the large model-based financial data warehouse intelligent SQL optimization method of claim 1, it is characterized in that: In S3, it specifically includes: Based on the multiple optimized execution plans generated by the big model, comprehensive evaluation is performed in combination with rich historical query performance data. The historical query performance data includes detailed information on the response time, resource consumption, and data read volume of different queries in the past. Use the performance prediction model to predict the performance of each execution plan under the current data status and system environment; The performance prediction model is built based on machine learning algorithms. It predicts the execution time and key performance parameters of resource requirements for each execution plan based on the relationship between features and performance indicators in historical data. Select the execution plan with the best predicted performance and monitor its execution efficiency in real time during the execution process. Use system monitoring tools to collect the actual execution time and resource usage data of the execution plan in real time, and compare and analyze them with the predicted values. If it is found that the actual execution efficiency deviates greatly from the predicted value, make timely adjustments or reselect the execution plan.

5. According to the big model-based financial data warehouse intelligent SQL optimization method of claim 1, it is characterized in that: In S4, it specifically includes: Dynamically adjust the allocation of computing resources according to the complexity of the query. For simple queries, reduce the usage of computing resources; for complex queries, intelligently increase the number of CPU cores and memory allocation resources; By evaluating the number of tables in the query statement, the complexity of the connection conditions, and the size of the data in real time, the resource allocation algorithm is used to determine the appropriate resource allocation plan; Select appropriate parallel processing methods based on the parallelism of the query. For queries with high parallelism, a parallel processing method based on data partitioning can be used to distribute query tasks to multiple computing nodes or threads for simultaneous execution. For queries with data dependencies, pipeline parallel processing is used to arrange the parallel execution order of each computing step to maximize the utilization of computing resources.

6. A financial data warehouse intelligent SQL optimization system based on a large model, characterized by: include: SQL parsing and preprocessing module, used to perform word segmentation, syntax analysis and logical structure analysis on SQL statements submitted by users, and identify keywords, table structures and query conditions in queries; The big model semantic understanding and optimization suggestion module is used to use the big model to perform semantic understanding and logical reasoning on the parsed SQL statements and automatically generate optimization suggestions; Dynamic execution plan generation and selection module, which is used to select the optimal execution plan based on the generated multiple optimized execution plans and combine historical query performance data, and monitor its execution efficiency in real time; The intelligent resource allocation and parallel execution module is used to reasonably allocate computing resources based on optimized SQL queries and adopt parallel processing to improve query performance.

7. The financial data warehouse intelligent SQL optimization system based on a large model according to claim 6 is characterized in that: SQL parsing and preprocessing modules include: The word segmentation unit is used to split the SQL statement submitted by the user into separate words or phrases; Syntax analysis unit, used to parse the syntax structure of SQL statements and generate syntax trees; Logical structure parsing unit, used to analyze the logical structure of SQL statements; The syntax tree building unit is used to build the parsed SQL statement into a syntax tree.

8. The financial data warehouse intelligent SQL optimization system based on a large model according to claim 7 is characterized in that: The large model semantic understanding and optimization suggestion module includes: The semantic understanding unit is used to use the big model to understand the SQL statements semantically and identify their intent and meaning; A deep learning algorithm unit, which is used to encode and parse SQL statements using deep learning algorithms; Optimization suggestion generation unit: used to generate index recommendations, query rewrites, and execution plan optimization suggestions based on semantic representations, combined with optimization rules and strategies.

9. The financial data warehouse intelligent SQL optimization system based on a large model according to claim 8 is characterized in that: The dynamic execution plan generation and selection module includes: The execution plan generation unit generates multiple optimized execution plans based on the optimization suggestions of SQL statements and large models; Historical query performance database, used to store historical query performance data, including query statements, execution plans, and execution time; Performance prediction unit, used to predict the performance of the execution plan and evaluate its execution efficiency and resource consumption; An execution plan selection unit is used to select the optimal execution plan by combining historical query performance data and performance prediction results; The real-time monitoring unit is used to monitor the execution efficiency of the execution plan in real time.

10. The financial data warehouse intelligent SQL optimization system based on a large model according to claim 8, characterized in that: The intelligent resource allocation and parallel execution module includes: A resource evaluation unit, used to calculate the complexity of the query statement according to its syntax and logical structure, and to evaluate the required computing resources; Resource allocation unit, used to reasonably allocate computing resources according to the complexity of the query and the usage of system resources; A parallel processing unit, used to select appropriate parallel processing according to the parallelism of the query; The resource monitoring and adjustment unit is used to monitor resource usage in real time and make dynamic adjustments based on actual needs.

Citation Information

Cited By

  • SQL (Structured Query Language) statement analysis optimization method and system based on large language model

    CN120353822A

  • A sql statement analysis and optimization method and system based on a large language model

    CN120353822B

  • SQL (Structured Query Language) optimization interaction method and device based on deep learning framework large model

    CN120508569A

  • Optimization suggestion determination method and device, storage medium and electronic device

    CN120561152A

  • Database dynamic query optimization and resource scheduling method, equipment and medium

    CN120849458A