Self-adaptive execution engine optimization method based on dynamic query

Abstract

The invention belongs to the technical field of database management, and particularly discloses a dynamic query-based adaptive execution engine optimization method, which comprises the following steps of: acquiring real-time multi-dimensional data of a database system; determining an optimal execution plan of the next time period based on the real-time multi-dimensional data; performing transaction consistency verification on the optimal execution plan; and under the condition that the verification is passed, seamlessly switching the current execution plan to the optimal execution plan. According to the method, on one hand, the problem of sudden increase of service flow or resource supply and demand mismatching in a low peak period can be avoided, the flexibility is improved, on the other hand, paths can be automatically switched in network fluctuation or node failure, and the labor cost is reduced.

Description

Technical Field

[0001] This application belongs to the field of database management technology, and more specifically, relates to an adaptive execution engine optimization method based on dynamic queries. Background Technology

[0002] Traditional database query optimization methods are generally based on pre-generated execution plan trees. On the one hand, this method can lead to database nodes experiencing resource overload and timeouts when faced with a surge in business traffic, while still occupying reserved resources during off-peak hours and unable to dynamically release them to other businesses, resulting in insufficient flexibility. On the other hand, it cannot automatically switch paths when there are network fluctuations or node failures, requiring manual monitoring by operations and maintenance personnel, which leads to high costs. Summary of the Invention

[0003] In view of the shortcomings of the existing technology, the purpose of this application is to provide an adaptive execution engine optimization method based on dynamic query, which aims to solve the problems of insufficient flexibility and high cost caused by static query optimization in traditional database query optimization methods.

[0004] To achieve the above objectives, in a first aspect, this application provides an adaptive execution engine optimization method based on dynamic queries, comprising: Collect real-time, multi-dimensional data from the database system; Based on the real-time multi-dimensional data, the optimal execution plan for the next time period is determined; Perform transaction consistency verification on the optimal execution plan; If the verification is successful, the current execution plan will be seamlessly switched to the optimal execution plan.

[0005] This application monitors multi-dimensional data from the database system in real time and adjusts the optimal execution plan for the next time period based on data changes. After verifying the transaction consistency of the optimal execution plan, it performs seamless switching. On the one hand, it can avoid the problem of resource supply and demand mismatch during business traffic surges or off-peak periods, improving flexibility. On the other hand, it can automatically switch paths when there are network fluctuations or node failures, reducing manual costs.

[0006] According to the adaptive execution engine optimization method based on dynamic query provided in this application, the real-time multi-dimensional data collection of the database system includes: The system collects real-time hardware layer data, real-time data layer data, and real-time query layer data from the database system. The hardware layer data includes central processing unit (CPU) utilization, memory page fault rate, number of input / output operations per second (IOPS) on the disk, and network latency. The data layer data includes the access frequency and lock contention intensity of data shards on each node. The query layer data includes operator execution time, intermediate result set size, and resource utilization.

[0007] This application achieves real-time reconstruction of the execution plan through three layers of data feedback, thereby improving the reliability of the execution plan.

[0008] According to the adaptive execution engine optimization method based on dynamic query provided in this application, determining the optimal execution plan for the next time period based on the real-time multi-dimensional data includes: Based on the real-time multi-dimensional data, multiple high-quality solutions are generated through a genetic algorithm. Multiple different preset optimization strategies are executed in the multiple high-quality solutions respectively, and strategy scores corresponding to the multiple different preset optimization strategies are obtained based on preset scoring rules; The optimization strategy with the highest strategy score will be used as the optimal execution plan for the next time period.

[0009] This application uses real-time data and combines genetic algorithms and reinforcement learning to generate the optimal execution plan for the next time period, solving the multi-objective trade-off problem in a distributed environment and improving the reliability of the execution plan.

[0010] According to the adaptive execution engine optimization method based on dynamic query provided in this application, the method further includes: Based on the real-time multi-dimensional data, the predicted system state value for the next time period is obtained through a prediction model. After switching to the optimal execution plan, the true system state value for the next time period is obtained; The prediction model is corrected based on the difference between the predicted value and the actual value.

[0011] According to the adaptive execution engine optimization method based on dynamic query provided in this application, the step of seamlessly switching the current execution plan to the optimal execution plan after successful verification includes: Adjust the thread pool size based on the optimal execution plan; The optimal execution plan is loaded in the background, while the current execution plan continues to run; Switch the current execution plan to the optimal execution plan at the transaction boundary.

[0012] This application adjusts the thread pool size based on the optimal execution plan to achieve resource supply and demand balance, and switches the current execution plan to the optimal execution plan at the transaction boundary to improve stability.

[0013] According to the adaptive execution engine optimization method based on dynamic query provided in this application, after seamlessly switching the current execution plan to the optimal execution plan upon successful verification, the method further includes: If an error occurs after the switch, the execution plan will be rolled back to the previous stable version.

[0014] This application rolls back to the previous stable version when the new execution plan encounters an anomaly, improving stability.

[0015] Secondly, this application provides an adaptive execution engine optimization device based on dynamic queries, comprising: The data acquisition module is used to acquire real-time, multi-dimensional data from the database system. The determination module is used to determine the optimal execution plan for the next time period based on the real-time multi-dimensional data. The verification module is used to verify the transaction consistency of the optimal execution plan; The switching module is used to seamlessly switch the current execution plan to the optimal execution plan if the verification is successful.

[0016] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the adaptive execution engine optimization method based on dynamic query described in the first aspect or any possible implementation thereof.

[0017] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the adaptive execution engine optimization method based on dynamic queries as described in the first aspect or any possible implementation of the first aspect.

[0018] Fifthly, this application provides a computer program product that, when run on a processor, causes the processor to execute the adaptive execution engine optimization method based on dynamic queries as described in the first aspect or any possible implementation of the first aspect.

[0019] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0020] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art: This application monitors multi-dimensional data from the database system in real time and adjusts the optimal execution plan for the next time period based on data changes. After verifying the transaction consistency of the optimal execution plan, it performs seamless switching. On the one hand, it can avoid the problem of resource supply and demand mismatch during business traffic surges or off-peak periods, improving flexibility. On the other hand, it can automatically switch paths when there are network fluctuations or node failures, reducing manual costs. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the adaptive execution engine optimization method based on dynamic query provided in an embodiment of this application; Figure 2 This is a schematic diagram of the dynamic optimization architecture provided in the embodiments of this application; Figure 3 This is a schematic diagram of the workflow of the dynamic optimization architecture provided in the embodiments of this application; Figure 4 This is a flowchart illustrating the genetic algorithm stage provided in an embodiment of this application; Figure 5 This is a flowchart illustrating the reinforcement learning phase provided in an embodiment of this application; Figure 6 This is a flowchart illustrating the rolling time-domain optimization stage provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of the adaptive execution engine optimization device based on dynamic query provided in the embodiments of this application; Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0025] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0026] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0027] First, let's introduce the following: Distributed query optimization technology addresses the performance bottleneck of cross-node queries in massive data scenarios, involving key technologies such as data sharding strategies, parallel computing frameworks, and network latency optimization.

[0028] The adaptive execution engine for Online Analytical Processing (OLAP) scenarios focuses on the high concurrency and low latency requirements of OLAP. It combines in-memory computing, vectorized execution, pre-aggregation and other technologies to optimize multidimensional analysis performance. By integrating columnar storage and dynamic materialized views, it solves the efficiency problem of traditional row storage engines in complex aggregation queries.

[0029] Multimodal resource collaborative management integrates mechanisms such as hardware resource awareness, distributed caching strategies, and elastic thread pools to achieve dynamic matching between query execution and underlying resources.

[0030] Traditional static query optimization is based on a pre-generated execution plan tree, which cannot dynamically detect changes in runtime resource status, leading to sudden service degradation and resource waste.

[0031] When business traffic surges (such as during e-commerce promotions or flash sales), the static optimizer cannot adjust the query plan based on real-time CPU / memory load, causing database nodes to time out due to resource overload. For example, during peak order periods, the payment system may refuse service due to CPU overload, directly resulting in transaction failures and revenue losses.

[0032] Static strategies continue to occupy reserved resources during off-peak hours, unable to be dynamically released to other business operations. For example, nighttime report generation tasks consume a large amount of memory, leading to insufficient memory for online business operations, forcing enterprises to expand their servers, increasing hardware costs by more than 30%.

[0033] Distributed query sharding technology reduces the pressure on the master database by sharding data, but it does not solve the problem of cross-node optimization for complex queries, resulting in low data analysis efficiency and decision delays.

[0034] Multi-table join queries (such as user behavior + order data) require scanning dozens of nodes. Due to the lack of optimization in cross-node transmission, hourly report generation is delayed. For example, the marketing department cannot obtain user conversion funnel data in real time, missing the best time to adjust campaigns.

[0035] When a node stores frequently accessed data (such as orders for popular products), complex queries concentrate on accessing that node, triggering a chain reaction of delays. A typical example: a logistics system experiences a backlog of order queries at a particular warehouse, slowing down the updating of waybill status across the entire network and leading to a surge in customer complaints.

[0036] Structured Query Language (SQL) syntax tree optimization reduces redundant code by refactoring SQL statements through the compiler, but it lacks a runtime dynamic adjustment mechanism, resulting in impaired business continuity and high operation and maintenance costs.

[0037] When there are network fluctuations or node failures, static execution plans cannot automatically switch paths. For example, if a bank's end-of-day batch processing task is interrupted due to a single node failure, manual intervention is required to restart it, resulting in settlement delays and violations of regulatory timeliness requirements.

[0038] Database administrators need to monitor and manually optimize queries 24 / 7. For example, in a financial case, the team handled over 200 performance alerts per month, with 60% of manpower dedicated to emergency adjustments, resulting in annual maintenance costs exceeding ten million.

[0039] To address the aforementioned shortcomings, this application provides an adaptive execution engine optimization method based on dynamic queries. Next, in conjunction with... Figures 1-7 The adaptive execution engine optimization method based on dynamic query provided in the embodiments of this application is introduced.

[0040] Figure 1 This is a flowchart illustrating the adaptive execution engine optimization method based on dynamic query provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps: Step 100: Collect real-time multi-dimensional data from the database system; Optionally, multi-dimensional data can be multi-dimensional operational data of the database system, such as CPU utilization, disk IOPS, etc.

[0041] Optionally, this application does not limit the method of collecting real-time multi-dimensional data, and real-time data can be automatically collected through data acquisition systems or other means.

[0042] Optionally, the collected data can be organized in a unified format, assigned metadata such as timestamps and data source identifiers, and a two-dimensional dynamic matrix can be constructed. The data can be a time series (e.g., 1 data point per second) and listed as monitoring variables (e.g., CPU, memory, fragmentation heat, lock contention intensity, etc.).

[0043] Optionally, abnormal events can be initially screened, such as by using built-in predefined rules (e.g., "CPU utilization consistently >90% for more than 5 seconds"), matching matrix data in real time, and triggering initial alerts (e.g., marking the high-load node SHARD_3).

[0044] Step 110: Based on real-time multi-dimensional data, determine the optimal execution plan for the next time period; Optionally, an optimal execution plan can be determined when abnormal events occur in real-time multi-dimensional data, or at regular intervals, such as every 5 seconds.

[0045] Optionally, adjustments can be made based on abnormal data in real-time multi-dimensional data to generate the optimal execution plan for the next time period.

[0046] Step 120: Verify the transaction consistency of the optimal execution plan; Transaction consistency is a crucial concept in computer science and technology, primarily used in database and distributed system scenarios to ensure the atomicity, consistency, isolation, and durability of transaction operations. Its core objective is to maintain data integrity during concurrent operations or system failures. By constraining the execution logic of transactions, it ensures that the final result of multi-step operations meets expectations, providing theoretical support for data correctness.

[0047] Step 130: If the verification passes, seamlessly switch the current execution plan to the optimal execution plan.

[0048] Optionally, after the optimal execution plan passes the transaction consistency verification, hot-swappable optimized instructions are generated.

[0049] Based on hot-swappable optimization instructions, the current execution plan can be seamlessly switched to the optimal execution plan.

[0050] Figure 2 This is a schematic diagram of the dynamically optimized architecture provided in the embodiments of this application, such as... Figure 2 As shown, in one embodiment of this application, three core modules are set up: a dynamic monitoring layer, an optimization decision engine, and an adaptive actuator, to achieve closed-loop control from environmental perception to execution optimization.

[0051] Figure 3 This is a schematic diagram of the workflow of the dynamic optimization architecture provided in the embodiments of this application, as shown below. Figure 3 As shown, the linkage process of each module strictly follows the closed-loop control logic of "perception → decision → execution → feedback", and the specific steps are as follows: 1a. After the database system starts running, the dynamic monitoring layer is continuously activated to collect multi-dimensional data, format data, build a real-time data matrix, and perform initial screening and early warning of abnormal events; 2a. The dynamic monitoring layer issues early warnings or periodically triggers the optimization decision engine to implement dynamic strategies, namely, generating execution plans and updating the strategy library; 3a. After receiving the hot-swappable optimization instructions generated by the optimization decision engine, the adaptive executor seamlessly switches to the optimal execution plan; 4a. After the optimal execution plan is executed, the monitoring layer collects actual performance data. Specifically, it can compare key indicators before and after optimization, such as response time, CPU volatility, and timeout rate, and generate performance reports. Successful strategies can also be stored in the strategy library and classified by scenario (such as OLAP complex queries / high concurrency online transaction processing). Furthermore, it can be continuously trained based on reinforcement learning models: taking environmental parameters (temperature, load) as input and outputting control parameters (voltage curve, adjustment step size) to improve the accuracy of future decisions.

[0052] The adaptive execution engine optimization method based on dynamic query provided in this application monitors multi-dimensional data of the database system in real time and adjusts the optimal execution plan for the next time period based on data changes. After verifying the transaction consistency of the optimal execution plan, it performs seamless switching. On the one hand, it can avoid the problem of resource supply and demand mismatch during business traffic surges or off-peak periods, improving flexibility. On the other hand, it can automatically switch paths when there are network fluctuations or node failures, reducing manual costs.

[0053] In some embodiments, step 100 specifically includes: The system collects real-time hardware layer data, real-time data layer data, and real-time query layer data from the database system. Hardware layer data includes CPU utilization, memory page fault rate, disk read / write operations per second (IOPS), and network latency. Data layer data includes access frequency and lock contention intensity for each node's data shards. Query layer data includes operator execution time, intermediate result set size, and resource utilization.

[0054] Optionally, CPU utilization can be sampled every 100ms.

[0055] Optionally, the lock contention intensity can be the percentage of lock wait time, etc.

[0056] Alternatively, hardware-level feedback can reduce cross-node memory access latency through Non-Uniform MemoryAccess (NUMA)-aware scheduling, as shown in the following formula:

[0057] In the formula, For task threads, Let be the memory access volume of task i on node j. To reduce the latency of access between NUMA nodes.

[0058] Optionally, the data layer feedback can be based on a columnar storage hot data prefetching algorithm, with the compression ratio improvement formula as follows:

[0059] in, For data type coefficients, text ≈0.9, image ≈0.4, For data sparsity, This is the efficiency constant of the compression algorithm.

[0060] Optionally, the query layer feedback can dynamically adjust the hybrid mode of volcano model and vectorized execution.

[0061] In some embodiments, step 110 specifically includes: Step 1101: Based on real-time multi-dimensional data, generate multiple high-quality solutions using a genetic algorithm; Step 1102: Execute multiple different preset optimization strategies in multiple high-quality solutions, and obtain the strategy scores corresponding to the multiple different preset optimization strategies based on preset scoring rules; Step 1103: Select the optimization strategy with the highest strategy score as the optimal execution plan for the next time period.

[0062] This application employs a hybrid dynamic optimization algorithm, combining genetic algorithms and deep reinforcement learning to generate the optimal execution plan for the next time period.

[0063] Figure 4 This is a flowchart illustrating the genetic algorithm stage provided in an embodiment of this application, as shown below. Figure 4 As shown, the core objective of the genetic algorithm is to quickly scan the solution space and narrow down the range of high-quality solutions through the biological logic of "generation → elimination → recombination". The algorithm flow is as follows: 1b. Generate diverse solutions: Randomly create hundreds of initial solutions (such as parameter combinations and path planning), covering a wide range of possibilities; 2b. Evaluation and Screening: Use simple rules (such as calculation time and cost) to score solutions, retain high-scoring solutions (top 20%), and eliminate inefficient solutions; 3b. Reorganization and Innovation: Piece together fragments of high-quality solutions (e.g., the first half of solution A + the second half of solution B) and randomly fine-tune details (e.g., ±10% parameter changes). 4b. Iterative loop: Repeat the above process until the scheme's performance is stable or the maximum number of iterations is reached.

[0064] Figure 5 This is a flowchart illustrating the reinforcement learning phase provided in an embodiment of this application, as shown below. Figure 5 As shown, if the genetic algorithm meets the stopping condition, it enters the reinforcement learning phase. The core objective of the reinforcement learning phase is to refine the strategy within the range of high-quality solutions defined by the genetic algorithm through a trial-and-error mechanism of "trying → scoring → optimizing". The process of this phase is as follows: 1c. Environmental awareness: Real-time monitoring of system status (such as load and data traffic), inheriting the optimal solution from the genetic stage; 2c. Strategy decision-making: If it is a low-risk environment, adopt the best historical strategy; if it is an unknown environment, try a new strategy and record the results. 3c. Feedback learning: Each action of the strategy is scored to obtain a strategy score. Specifically, successful actions are rewarded with bonus points, such as 2 points for corresponding speed increase, while failed actions are penalized with deduction points, such as 5 points for exceeding the time limit. 4c. Continuous adaptation: Cycle through "perception → decision → feedback" until the task is completed or dynamic changes are adapted, such as a surge in traffic.

[0065] In some embodiments, the method further includes: Step 140: Based on real-time multi-dimensional data, obtain the predicted system state value for the next time period through a prediction model; Step 150: After switching to the optimal execution plan, obtain the actual system state value for the next time period; Step 160: Based on the difference between the predicted value and the actual value, the prediction model is corrected.

[0066] Figure 6 This is a flowchart illustrating the rolling time-domain optimization stage provided in an embodiment of this application, as shown below. Figure 6 As shown, after the reinforcement learning task is completed, the rolling time-domain optimization phase begins. The core objective of the rolling time-domain optimization phase is to continuously optimize decisions within a finite time window through a rolling process of "prediction → execution → feedback → adjustment". The process of this phase is as follows: 1d. Predict future state: Based on current data, predict the system state (such as energy consumption and load) in the short term (such as one millisecond / minute) using a predictive model. 2d. Execute the optimal action: Select the optimal execution plan for the current time period (such as adjusting server resource allocation); 3D, collecting feedback data: Real-time acquisition of motion effects, i.e., the actual value of the system state, and comparison with the predicted value; 4d. Roll to the next time period: Use feedback data to correct the prediction model and move to the next time period for re-optimization.

[0067] In some embodiments, step 130 specifically includes: Step 1301: Adjust the thread pool size based on the optimal execution plan; Step 1302: The optimal execution plan is loaded in the background, while the current execution plan continues to run; Step 1303: Switch the current execution plan to the optimal execution plan at the transaction boundary.

[0068] After obtaining the optimal execution plan, the thread pool size is dynamically adjusted based on the optimal execution plan to avoid high-load nodes.

[0069] During hot swapping, a double buffering mechanism is adopted. On the one hand, while the optimal execution plan is loaded in the background, the old plan continues to run. On the other hand, atomic switching is performed at the transaction boundary to ensure no interruption (e.g., TPC-H Q17 switching time <8ms).

[0070] Optionally, the lock granularity can be dynamically switched based on the transaction conflict rate, as shown in the following formula:

[0071] in, For concurrent transactions, For the percentage of write operations, , The coefficient is updated based on historical conflict rates.

[0072] In some embodiments, after step 130, the method further includes: If an error occurs after the switch, the execution plan will be rolled back to the previous stable version.

[0073] If an anomaly occurs after the switch (such as an increase in deadlock rate), the system will automatically roll back to the previous stable version and trigger the decision engine to re-optimize.

[0074] In one embodiment of this application, the adaptive execution engine optimization method based on dynamic query provided in this application and the traditional static optimization method are subjected to TPC-H benchmark tests, and the comparison results are shown in Table 1 below: Table 1 Experimental Results

[0075] Experiments show that this application achieves intelligent adaptive optimization of the database kernel through hardware-aware dynamic resource scheduling, hybrid optimization algorithms and multi-level feedback mechanisms. Compared with the traditional solution TPC-H Q17, the response time of complex queries is reduced by 42%, the CPU utilization fluctuation range is reduced to ±15%, sudden resource exhaustion is avoided, fault nodes are automatically avoided, and the query failure rate is reduced by 76%.

[0076] In one embodiment of this application, taking a promotional period on an e-commerce platform as an example, the following workloads are executed simultaneously: real-time order statistics (OLTP), user behavior analysis (OLAP), and inventory forecasting (machine learning inference). The data preparation is as follows: CREATE TABLE orders ( id BIGINT PRIMARY KEY SHARD KEY, user_id INT, amount DECIMAL(10,2), ts TIMESTAMP ) DISTRIBUTED BY HASH(id) INTO 8 SHARDS; Trigger dynamic optimization: When the CPU utilization of SHARD_3 is detected to be consistently >90%, queries involving this shard will be automatically rerouted to the backup shard SHARD_3_replica, and the join order will be adjusted: change orders JOIN users to users JOIN orders.

[0077] The results are shown in Table 2 below: Table 2 Effect Verification

[0078] Figure 7 This is a schematic diagram of the structure of the adaptive execution engine optimization device based on dynamic query provided in the embodiments of this application, as shown below. Figure 7 As shown, the device includes a data acquisition module 710, a determination module 720, a verification module 730, and a switching module 740, wherein: The acquisition module 710 is used to acquire real-time multi-dimensional data from the database system. The determination module 720 is used to determine the optimal execution plan for the next time period based on real-time multi-dimensional data; Verification module 730 is used to verify the transaction consistency of the optimal execution plan; The switching module 740 is used to seamlessly switch the current execution plan to the optimal execution plan if the verification is successful.

[0079] Based on the methods in the above embodiments, Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown in the illustration, this application provides an electronic device that may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840. The processor 810, communications interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions from the memory 830 to execute the adaptive execution engine optimization method based on dynamic query described in the above embodiment.

[0080] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the adaptive execution engine optimization method based on dynamic query described in the various embodiments of this application.

[0081] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the adaptive execution engine optimization method based on dynamic query in the above embodiments.

[0082] Based on the methods in the above embodiments, this application provides a computer program product that, when running on a processor, causes the processor to execute the adaptive execution engine optimization method based on dynamic query in the above embodiments.

[0083] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0084] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0085] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0086] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0087] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An adaptive execution engine optimization method based on dynamic query, characterized in that, include: Collect real-time, multi-dimensional data from the database system; Based on the real-time multi-dimensional data, the optimal execution plan for the next time period is determined; Perform transaction consistency verification on the optimal execution plan; If the verification is successful, the current execution plan will be seamlessly switched to the optimal execution plan.

2. The adaptive execution engine optimization method based on dynamic query according to claim 1, characterized in that, The real-time multi-dimensional data collected by the database system includes: The system collects real-time hardware layer data, real-time data layer data, and real-time query layer data from the database system. The hardware layer data includes CPU utilization, memory page fault rate, disk read / write operations per second (IOPS), and network latency. The data layer data includes the access frequency and lock contention intensity of data shards on each node. The query layer data includes operator execution time, intermediate result set size, and resource utilization.

3. The adaptive execution engine optimization method based on dynamic query according to claim 1, characterized in that, The step of determining the optimal execution plan for the next time period based on the real-time multi-dimensional data includes: Based on the real-time multi-dimensional data, multiple high-quality solutions are generated through a genetic algorithm. Multiple different preset optimization strategies are executed in the multiple high-quality solutions respectively, and strategy scores corresponding to the multiple different preset optimization strategies are obtained based on preset scoring rules; The optimization strategy with the highest strategy score will be used as the optimal execution plan for the next time period.

4. The adaptive execution engine optimization method based on dynamic query according to claim 1, characterized in that, The method further includes: Based on the real-time multi-dimensional data, the predicted system state value for the next time period is obtained through a prediction model. After switching to the optimal execution plan, the true system state value for the next time period is obtained; The prediction model is corrected based on the difference between the predicted value and the actual value.

5. The adaptive execution engine optimization method based on dynamic query according to claim 1, characterized in that, The step of seamlessly switching the current execution plan to the optimal execution plan upon successful verification includes: Adjust the thread pool size based on the optimal execution plan; The optimal execution plan is loaded in the background, while the current execution plan continues to run; Switch the current execution plan to the optimal execution plan at the transaction boundary.

6. The adaptive execution engine optimization method based on dynamic query according to claim 1, characterized in that, After seamlessly switching the current execution plan to the optimal execution plan upon successful verification, the method further includes: If an error occurs after the switch, the execution plan will be rolled back to the previous stable version.

7. An adaptive execution engine optimization device based on dynamic query, characterized in that, include: The data acquisition module is used to acquire real-time, multi-dimensional data from the database system. The determination module is used to determine the optimal execution plan for the next time period based on the real-time multi-dimensional data. The verification module is used to verify the transaction consistency of the optimal execution plan; The switching module is used to seamlessly switch the current execution plan to the optimal execution plan if the verification is successful.

8. An electronic device, characterized in that, include: At least one memory for storing computer programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the adaptive execution engine optimization method based on dynamic query as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is run on the processor, it causes the processor to perform the adaptive execution engine optimization method based on dynamic query as described in any one of claims 1-6.

10. A computer program product, characterized in that, When the computer program product is run on a processor, the processor performs the adaptive execution engine optimization method based on dynamic query as described in any one of claims 1-6.

Images