Database query prediction and load scheduling method and device

Abstract

The invention provides a database query prediction and load scheduling method and device. The method comprises the following steps: obtaining a plurality of query tasks corresponding to a database system; respectively inputting the plurality of query tasks into a preset memory prediction model, and obtaining memory prediction results which are output by the memory prediction model and respectively correspond to the plurality of query tasks; the memory prediction result is a memory demand prediction value; performing load scheduling on the plurality of query tasks based on the memory prediction result and a dynamic scheduling strategy to obtain load scheduling information, and performing memory allocation processing on the plurality of query tasks based on the load scheduling information; wherein the dynamic scheduling strategy is a load scheduling strategy based on memory perception of the database system; the load scheduling information comprises a query execution sequence of a plurality of query tasks. According to the method provided by the invention, the memory prediction precision and the load scheduling efficiency of the query task can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and in particular, to a database query prediction and load scheduling method and apparatus. In addition, the present invention also relates to an electronic device, a non-transitory computer-readable storage medium, and a computer program product. Background Art

[0002] In recent years, with the rapid development of modern database systems, memory management has gradually become one of the key factors affecting query performance. In a high-performance computing environment, the memory consumption of queries and the optimization of scheduling strategies directly determine the overall efficiency of the system. However, existing query memory prediction methods usually rely on empirical rules or static cost estimation models, and these methods are difficult to adapt to complex multi-user scenarios and high-dynamic load environments. Therefore, researching how to effectively predict the memory usage of queries and design a highly adaptable scheduling strategy has become an important topic in the database field. Existing query memory prediction mainly relies on static analysis or rule models. Specifically, a database management system (DBMS) usually generates a query plan during the query compilation phase and estimates the memory requirements of the query by estimating the costs of operators (such as I / O costs and CPU time). Although these methods are simple, they rely on the assumption that the query environment is stable and the query plan is accurate, and it is difficult to handle complex data distributions or variable workloads. For example, PostgreSQL uses a cost model to estimate the memory requirements of each operator, but this method usually ignores the actual data characteristics at runtime. In addition, inaccurate memory prediction may lead to improper resource allocation, further reducing query performance. For example, excessive memory allocation will result in resource waste, while too little allocation may cause overflow operations (such as frequent creation of disk temporary files), increasing the query latency and I / O costs.

[0003] In addition, in a multi-user environment, the load scheduling of a database system mainly solves the problems of resource allocation and execution order of query tasks. The current mainstream scheduling strategies are generally divided into priority-based scheduling, cost-based scheduling, and simple round-robin scheduling. Among them, cost-based scheduling attempts to dynamically adjust the priorities of tasks according to the resource requirements of queries and the current load situation of the system. However, this method usually relies on accurate cost estimation, and traditional cost models are difficult to handle the comprehensive trade-off of multi-dimensional resource requirements (such as memory, CPU, I / O, etc.). In recent years, with the rise of real-time analysis and online transaction processing (HTAP) scenarios, the types of loads faced by database systems have become more complex, and the scheduling difficulty has increased further. For example, in a high-concurrency scenario, multiple query tasks may compete for memory resources, resulting in resource contention and performance bottlenecks. At the same time, the memory requirements of different query tasks vary significantly, and simple scheduling algorithms cannot fully utilize the resources of the database system. In addition, in a cloud computing environment, the dynamic scaling of database system resources and the optimization of rental costs also pose higher requirements for scheduling strategies. Therefore, how to provide a more efficient database query prediction and load scheduling scheme has become an urgent technical problem to be solved currently. Summary of the Invention

[0004] The present invention provides a database query prediction and load scheduling method to solve the defect in the prior art that the limitations of the database query prediction and load scheduling scheme are relatively high, resulting in poor processing efficiency of query tasks.

[0005] The present invention provides a database query prediction and load scheduling method, including: Obtaining a plurality of query tasks corresponding to the database system; Inputting the plurality of query tasks into a preset memory prediction model respectively to obtain memory prediction results corresponding to the plurality of query tasks output by the memory prediction model; wherein, the memory prediction model; wherein, the memory prediction model is a heterogeneous graph neural network model trained based on a query plan and a memory prediction result label corresponding to the query plan respectively; the query plan is a plan corresponding to the query task; the memory prediction result is a memory requirement prediction value; Performing load scheduling on the plurality of query tasks based on the memory prediction results and a dynamic scheduling strategy to obtain load scheduling information, and performing memory allocation processing on the plurality of query tasks based on the load scheduling information; wherein, the dynamic scheduling strategy is a load scheduling strategy based on memory awareness of the database system; the load scheduling information includes the query execution order of the plurality of query tasks.

[0006] According to the database query prediction and load scheduling method of the present invention, before obtaining a plurality of query tasks corresponding to the database system, it further includes: Obtain a query plan corresponding to the database system; Perform a graph structure representation on the query plan to obtain a sample graph data structure corresponding to the query plan; based on the sample graph data structure, obtain an initial memory prediction model; Based on the query plan and the memory prediction result labels corresponding to the query plan, obtain a training set and a validation set; wherein, the memory prediction result labels are memory requirement prediction values respectively corresponding to the query plan; Based on the training set and the validation set, perform model training and validation on the initial memory prediction model to obtain a memory prediction model.

[0007] According to the database query prediction and load scheduling method of the present invention, the performing a graph structure representation on the query plan to obtain a graph data structure corresponding to the query plan includes: Define the operators in the query plan as nodes in the graph data structure; wherein, the attributes of the nodes include the type of the operator, the estimated intermediate result size, and the data distribution; Define the data flow relationship between the operators in the query plan as the edges constituting the graph data structure; wherein, the attributes of the edges of the graph data structure include the connection condition and the number of rows of data transmission; Based on the nodes in the graph data structure and the edges of the graph data structure, obtain the corresponding graph data structure.

[0008] According to the database query prediction and load scheduling method of the present invention, the inputting the multiple query tasks into a preset memory prediction model respectively to obtain memory prediction results corresponding to the multiple query tasks output by the memory prediction model includes: Perform feature representation on the multiple query tasks to obtain a graph data structure of the multiple query tasks; Extract features of the nodes in the graph data structure through the memory prediction model, and perform multi-layer propagation and aggregation on the node features to obtain memory prediction results corresponding to the multiple query tasks respectively.

[0009] According to the database query prediction and load scheduling method of the present invention, the performing load scheduling on the multiple query tasks based on the memory prediction results and a dynamic scheduling strategy to obtain load scheduling information, and performing memory allocation processing on the multiple query tasks based on the load scheduling information specifically includes: Model the multiple query tasks to obtain corresponding multiple query task models; Perform dynamic priority sorting on the multiple query task models based on the memory prediction results and the dynamic scheduling strategy to obtain the query execution order corresponding to the multiple query task models; Obtain the memory usage information of the database system; perform memory allocation processing on the multiple query tasks based on the memory usage information of the database system and the query execution order corresponding to the multiple query task models.

[0010] According to the database query prediction and load scheduling method of the present invention, after performing memory allocation processing on the multiple query tasks, it further includes: Obtain the latest memory usage information of the database system, update the dynamic scheduling policy based on the new memory usage information, and obtain a new dynamic scheduling policy; Perform load scheduling on the multiple query tasks input in the next memory prediction cycle based on the memory prediction result of the next memory prediction cycle and the new dynamic scheduling policy, and obtain new load scheduling information; Perform memory allocation processing on the multiple query tasks input in the next memory prediction cycle based on the new load scheduling information.

[0011] The present invention also provides a database query prediction and load scheduling device, including: A query task acquisition unit, configured to acquire multiple query tasks corresponding to a database system; A memory prediction processing unit, configured to input the multiple query tasks into a preset memory prediction model respectively, and obtain memory prediction results corresponding to the multiple query tasks output by the memory prediction model; wherein, the memory prediction model; wherein, the memory prediction model is a heterogeneous graph neural network model trained based on a query plan and memory prediction result labels respectively corresponding to the query plan; the query plan is a plan corresponding to the query task; the memory prediction result is a memory demand prediction value; A load scheduling processing unit, configured to perform load scheduling on the multiple query tasks based on the memory prediction result and a dynamic scheduling policy, obtain load scheduling information, and perform memory allocation processing on the multiple query tasks based on the load scheduling information; wherein, the dynamic scheduling policy is a load scheduling policy based on memory awareness of the database system; the load scheduling information includes the query execution order of multiple query tasks.

[0012] According to the database query prediction and load scheduling device of the present invention, before obtaining multiple query tasks corresponding to a database system, it further includes: a model training unit; The model training unit is specifically configured to: Obtain a query plan corresponding to a database system; Perform a graph structure representation on the query plan to obtain a sample graph data structure corresponding to the query plan; based on the sample graph data structure, obtain an initial memory prediction model; Obtain a training set and a validation set based on the query plan and the memory prediction result tags corresponding to the query plan; wherein, the memory prediction result tags are memory requirement prediction values respectively corresponding to the query plan. Based on the training set and the validation set, perform model training and validation on the initial memory prediction model to obtain a memory prediction model.

[0013] According to the database query prediction and load scheduling device of the present invention, the graph structure representation of the query plan to obtain the graph data structure corresponding to the query plan includes: Define the operators in the query plan as nodes in the graph data structure; wherein, the attributes of the nodes include the type of the operator, the estimated size of the intermediate result, and the data distribution. Define the data flow relationship between the operators in the query plan as the edges constituting the graph data structure; wherein, the attributes of the edges of the graph data structure include the connection condition and the number of rows of data transmission. Based on the nodes in the graph data structure and the edges of the graph data structure, obtain the corresponding graph data structure.

[0014] According to the database query prediction and load scheduling device of the present invention, the memory prediction processing unit is specifically used for: Perform feature representation on the multiple query tasks to obtain the graph data structure of the multiple query tasks. Extract features from the nodes in the graph data structure through the memory prediction model, and perform multi-layer propagation and aggregation on the node features to obtain the memory prediction results respectively corresponding to the multiple query tasks.

[0015] According to the database query prediction and load scheduling device of the present invention, the load scheduling processing unit is specifically used for: Model the multiple query tasks to obtain corresponding multiple query task models. Based on the memory prediction results and the dynamic scheduling strategy, perform dynamic priority sorting on the multiple query task models to obtain the query execution order corresponding to the multiple query task models. Obtain the memory usage information of the database system; based on the memory usage information of the database system and the query execution order corresponding to the multiple query task models, perform memory allocation processing on the multiple query tasks.

[0016] According to the database query prediction and load scheduling device of the present invention, after performing memory allocation processing on the multiple query tasks, it further includes: The dynamic scheduling strategy update processing unit is used for: Obtain the latest memory usage information of the database system, update the dynamic scheduling policy based on the new memory usage information, and obtain a new dynamic scheduling policy; Perform load scheduling on multiple query tasks input within the next memory prediction period based on the memory prediction result of the next memory prediction period and the new dynamic scheduling policy to obtain new load scheduling information; Perform memory allocation processing on multiple query tasks input within the next memory prediction period based on the new load scheduling information.

[0017] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the database query prediction and load scheduling method described in any one of the above.

[0018] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the database query prediction and load scheduling method described in any one of the above.

[0019] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the database query prediction and load scheduling method described in any one of the above.

[0020] The database query prediction and load scheduling method provided by the present invention obtains multiple query tasks corresponding to a database system, inputs the multiple query tasks into a preset memory prediction model respectively, obtains memory prediction results corresponding to the multiple query tasks output by the memory prediction model respectively, and performs load scheduling on the multiple query tasks based on the memory prediction results and a dynamic scheduling policy to obtain load scheduling information, and performs memory allocation processing on the multiple query tasks based on the load scheduling information. The dynamic scheduling policy is a load scheduling policy based on memory awareness of the database system; the load scheduling information includes the query execution order of multiple query tasks, and it can effectively improve the memory prediction accuracy and load scheduling efficiency of query tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0022] Figure 1It is a schematic flowchart of the database query prediction and load scheduling method provided by the present invention.

[0023] Figure 2 It is a schematic framework flowchart of the MemQ query memory prediction method provided by the present invention.

[0024] Figure 3 It is a schematic diagram of the dynamic scheduling strategy provided by the present invention.

[0025] Figure 4 It is a schematic diagram of the accuracy of cross-load prediction provided by the present invention.

[0026] Figure 5 It is a schematic diagram of the memory occupancy in load scheduling provided by the present invention.

[0027] Figure 6 It is a schematic structural diagram of the database query prediction and load scheduling device provided by the present invention.

[0028] Figure 7 It is a schematic structural diagram of the electronic device provided by the present invention. Detailed implementation manners

[0029] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without creative efforts shall fall within the protection scope of the present invention.

[0030] In recent years, as a deep learning method for processing unstructured data, the graph neural network (GNN) has gradually been introduced into the field of database systems. Query plans usually have a tree or graph structure, which highly coincides with the characteristics of graph neural networks that are good at processing graph data. By representing the query plan in the form of nodes and edges, the graph neural network can effectively extract the structural information and context relationships in the query plan, thereby improving the expressive ability of the memory prediction model. For example, in tasks such as index selection, cost estimation, and query optimization, the graph neural network has shown higher accuracy than traditional methods. In the memory prediction task, factors such as the operators, data sources, and filtering conditions of the query plan jointly determine the memory requirements. Existing methods are difficult to fully model the complex relationships between these factors, while the graph neural network can capture the global and local characteristics in the query plan through iterative graph structure learning, providing the possibility for more accurate memory prediction.

[0031] Generally speaking, the existing memory prediction and load scheduling methods face challenges such as insufficient accuracy, poor adaptability, and difficulty in coping with dynamic environments. The method based on graph neural network, with its powerful structured modeling ability, provides a new solution for query memory prediction and load scheduling. By constructing a more accurate memory prediction model and combining dynamic scheduling strategies, the performance and resource utilization rate of the database system in complex load scenarios can be significantly improved. For this reason, the present invention provides a database query prediction and load scheduling method and device. The following combines Figures 1-7 to describe the database query prediction and load scheduling method and device of the present invention, and details its embodiments.

[0032] As Figure 1 shown, it is one of the flow schematic diagrams of the database query prediction and load scheduling method provided by the present invention, and the specific implementation process includes the following steps: Step 101, obtain multiple query tasks corresponding to the database system.

[0033] In the embodiment of the present invention, before executing this step, it is necessary to pre-construct and train a model to obtain a memory prediction model that meets the application conditions. Specifically, first, it is necessary to obtain the query plan corresponding to the database system, perform a graph structure representation on the query plan, and obtain the sample graph data structure corresponding to the query plan; based on the sample graph data structure, obtain an initial memory prediction model; based on the query plan and the memory prediction result label corresponding to the query plan, obtain a training set and a validation set; where the memory prediction result label is the memory requirement prediction value corresponding to the query plan respectively; finally, based on the training set and the validation set, perform model training and verification on the initial memory prediction model to obtain a memory prediction model.

[0034] Among them, in the process of representing the query plan in a graph structure to obtain the graph data structure corresponding to the query plan, the operators in the query plan can be defined as nodes in the graph data structure; among them, the attributes of the nodes include the type of the operator, the estimated size of the intermediate result, and the data distribution, etc.; the data flow relationship between the operators in the query plan is defined as the edge constituting the graph data structure; among them, the attributes of the edge of the graph data structure include the connection condition and the number of rows of data transmission, etc.; based on the nodes in the graph data structure and the edges of the graph data structure, the corresponding graph data structure is obtained. Specifically, in the process of constructing the graph structure representation of the query plan, it includes: Step 1-1: When defining nodes, the operators in the query plan (such as scan, join, sort) are defined as nodes in the graph, and the node attributes include the operator type, the estimated size of the intermediate result, the data distribution, etc. Step 1-2: When defining edges, that is, the data flow relationship between operators constitutes the edges of the graph, and the edge attributes include the connection condition, the number of rows of data transmission, etc. Step 1-3: When generating the graph representation (that is, obtaining the graph data structure), use a graphical tool or custom code to parse the query plan into the form of nodes and edges, and generate the corresponding graph data structure.

[0035] In addition, in the process of designing the graph neural network model (that is, obtaining the initial memory prediction model based on the sample graph data structure), it specifically includes: Step 2-1: When inputting features, assign initial features to each node and edge. The initial features of the node include the attribute vector of the operator, and the initial features of the edge include the relevant information of the data flow. Step 2-2: When designing the architecture of the initial memory prediction model, adopt a graph neural network architecture based on the message passing mechanism. Take the graph isomorphism network (GIN) as an example. The network structure includes multiple layers of graph convolution, which are used to extract the high-level features of the nodes and the whole graph. Step 2-3: When outputting the target, the output of the memory prediction model is the predicted value of the memory requirement of the query plan.

[0036] In addition, in the process of preparing training data (that is, obtaining the training set and the validation set), it specifically includes: Step 3-1: When collecting the query plan and memory data, extract a large number of query plans (that is, query plans) from the database system, and execute to collect the actual memory usage data (that is, the memory prediction result labels corresponding to the query plans). Step 3-2: In feature normalization, normalize the collected query plan features and memory data to adapt to the training of the deep learning model. Step 3-3: When constructing the training set and the validation set, randomly divide the collected data into the training set and the validation set for the training and performance evaluation of the initial memory prediction model.

[0037] Further, in the process of training the initial graph neural network model, it specifically includes: Step 4-1: In the definition of the loss function, the Mean Squared Error (MSE) is used as the loss function to measure the deviation between the predicted value and the true memory value. Step 4-2: In the training of the initial graph neural network model, the Stochastic Gradient Descent (SGD) or Adam optimizer is used to minimize the loss function and iteratively update the model parameters. Step 4-3: In the training and validation of the graph neural network model, the prediction performance of the model is evaluated through the validation set, and hyperparameters (such as the learning rate and the number of nodes in the hidden layer) are adjusted to improve the model accuracy.

[0038] Step 102: Input the multiple query tasks into a preset memory prediction model respectively, and obtain the memory prediction results corresponding to the multiple query tasks output by the memory prediction model; wherein, the memory prediction model; wherein, the memory prediction model is a heterogeneous graph neural network model trained based on the query plan and the memory prediction result labels corresponding to the query plan respectively; the query plan is a plan corresponding to the query task; and the memory prediction result is the memory requirement prediction value.

[0039] In the embodiment of the present invention, the multiple query tasks can be feature-represented to obtain the graph data structure of the multiple query tasks; the memory prediction model is used to extract features from the nodes in the graph data structure, and perform multi-layer propagation and aggregation on the node features to obtain the memory prediction results corresponding to the multiple query tasks respectively.

[0040] The query memory prediction method based on the graph neural network (GNN) model of the present invention accurately predicts the peak memory requirement of the query by analyzing the structural features in the query plan. For example Figure 2As shown in the figure, its specific technical implementation includes the following steps: First, query plan feature representation: (1) The query plan is modeled as a graph data structure, and the nodes in the graph data structure include operation nodes (such as Hash Join, Index Scan) and data nodes (such as tables, columns). (2) Each node in the graph data structure contains category information (represented by one-hot encoding) and numerical features (such as operation cost, number of rows, table size), and is preprocessed by combining normalization and logarithmic transformation. Then, perform node feature embedding: (1) Each node is embedded through a graph neural network to generate a high-dimensional vector representation. (2) Feature aggregation is divided into two parts: The bottom-layer nodes aggregate the information of tables and columns through bottom-up message passing, and the upper-layer nodes obtain the overall features of the query plan through global aggregation. Finally, query memory prediction: (1) Through the graph neural network, multi-layer propagation and aggregation of node features are performed to generate the memory prediction value of the entire query plan. (2) The training of the model is optimized by minimizing the mean square error (MSE) loss function to ensure the accuracy and robustness of memory prediction.

[0041] Table 1: Accuracy effect of the memory prediction model trained on a single dataset As shown in Table 1 and Figure 4 As shown, through comparative analysis, it can be seen that the query memory prediction accuracy of the present invention has been significantly improved by the query memory prediction method based on the graph neural network (GNN) model, providing a reliable basis for subsequent load scheduling.

[0042] Step 103: Perform load scheduling on the multiple query tasks based on the memory prediction result and the dynamic scheduling policy to obtain load scheduling information, and perform memory allocation processing on the multiple query tasks based on the load scheduling information; wherein, the dynamic scheduling policy is a load scheduling policy based on memory awareness of the database system; the load scheduling information includes the query execution order of the multiple query tasks.

[0043] In the embodiment of the present invention, the multiple query tasks can be modeled to obtain corresponding multiple query task models; perform dynamic priority sorting on the multiple query task models based on the memory prediction result and the dynamic scheduling policy to obtain the query execution order corresponding to the multiple query task models; obtain the memory usage information of the database system; and perform memory allocation processing on the multiple query tasks based on the memory usage information of the database system and the query execution order corresponding to the multiple query task models.

[0044] Specifically, in the process of optimizing query execution in combination with the dynamic scheduling strategy, it includes: Step 5-1: Dynamic scheduling strategy design. According to the memory requirements predicted by the memory prediction model, dynamic priority sorting is performed on query tasks, and queries that efficiently utilize memory are preferentially executed. Step 5-2: Resource allocation in a multi-query environment. The memory usage of the database system is monitored in real time. Combining the scheduling strategy, queries are submitted within the available memory range to avoid resource contention. The greedy algorithm is adopted to preferentially allocate memory to queries that occupy a large amount of memory. Step 5-3: Dynamic scheduling adjustment. During the running process, the prediction result of the memory prediction model is adjusted according to the actual memory usage information of the query, and the load scheduling strategy is dynamically updated. That is, after performing memory allocation processing on the multiple query tasks, it further includes: obtaining the latest memory usage information of the database system, updating the dynamic scheduling strategy based on the new memory usage information to obtain a new dynamic scheduling strategy; performing load scheduling on the multiple query tasks input in the next memory prediction cycle based on the memory prediction result of the next memory prediction cycle and the new dynamic scheduling strategy to obtain new load scheduling information; performing memory allocation processing on the multiple query tasks input in the next memory prediction cycle based on the new load scheduling information.

[0045] Further, in the process of system integration and evaluation, it specifically includes: Step 6-1: System integration. Integrate the graph neural network model with the database scheduling module to form a complete query optimization framework. Step 6-2: Verification. It is carried out on benchmark datasets such as TPC-H or TPC-DS to compare the differences in memory prediction accuracy and query performance between the method of the present invention and traditional methods.

[0046] That is, based on the memory prediction result, the present invention further proposes a memory-aware load scheduling strategy, which optimizes the query execution order and resource allocation of the database system, as Figure 3 shown. Each query task is represented as a rectangle. The length of the rectangle represents the memory requirement size of the query, and the width of the rectangle represents the execution time of the query. The load scheduling problem is equivalent to placing several query rectangles into Figure 3Within the range shown on the right. The process of specifically implementing the dynamic load policy includes the following steps: 1. Conduct query task modeling: (1) Each query task is modeled as a rectangle, where the width represents the execution time of the query and the height represents the memory requirement. (2) By arranging the combination of rectangles, the resource usage situation during the concurrent execution of multiple tasks is simulated. 2. Execute the scheduling optimization policy (i.e., the dynamic scheduling policy). Specifically, a First-Fit Decreasing (FFD) algorithm is proposed to dynamically adjust the query execution order. The FFD algorithm preferentially allocates query tasks with larger memory requirements to ensure the efficient use of memory resources. (2) In each step of scheduling, the available memory of the database system is updated in real time according to the memory prediction result, and it is judged whether a new query task can be scheduled. If the predicted result of the memory requirement of the new query task exceeds the remaining memory of the database system, the execution is delayed to avoid resource conflicts. 3. Load balancing mechanism. (1) The present invention designs a priority-based load balancing mechanism to mix and schedule query tasks with low memory requirements and query tasks with high memory requirements, reducing the resource competition among multiple query tasks. (2) By dynamically adjusting the task order and balancing the memory allocation, the global optimum of the scheduling process is achieved. Through the load scheduling policy of the present invention, the overall efficiency of query execution and the system stability are significantly improved.

[0047] In addition, the present invention also proposes a memory prediction method with strong generalization ability for different load scenarios and database systems, which is particularly outstanding in cross-load prediction. The specific process includes the following steps: 1. Cross-load prediction ability (1) Based on the heterogeneous graph neural network model, it can adapt to different types of query plans and extract the general features of tasks. (2) Through feature transfer and global aggregation, the model can maintain high prediction accuracy under different loads. 2. Prediction performance evaluation: (1) Multiple evaluation metrics are adopted, including Mean Relative Error (MRE), Root Mean Square Error (RMSE), and QError (median, mean, 95th percentile), etc., for comprehensive evaluation. (2) It shows that the median of QError between different loads of the model of the present invention is significantly lower than that of the traditional model, verifying its superior cross-load prediction ability, as Figure 3 shown. Setting 1, 2, 3 respectively represent different training sets and test sets.

[0048] As shown in Table 2 and Figure 5As shown in the figure, the effectiveness of the memory-aware scheduling strategy of the present invention in high-concurrency query scenarios is verified through comparative analysis, which is mainly manifested in the following advantages: 1. Memory usage optimization: (1) Under the default strategy, the memory usage of query tasks fluctuates greatly, exceeding the memory limit multiple times and frequently triggering the use of swap space. (2) After adopting the FFD strategy of the present invention, the memory usage curve becomes stable, and the usage amount of the system swap space is significantly reduced, avoiding the performance degradation caused by memory competition. 2. Performance improvement: (1) In an environment with a limited memory of 3GB, the scheduling strategy of the present invention reduces the total execution time of 900 query tasks from 986 seconds to 439 seconds. (2) The number of retry queries is reduced from multiple retries under the default strategy to nearly 0, significantly improving the execution efficiency and stability of tasks. 3. Enhanced system stability: (1) By combining memory prediction and scheduling optimization, the database system operates efficiently under high-concurrency query tasks, avoiding query failures or system crashes caused by insufficient memory. (2) The flexibility of the dynamic scheduling strategy makes it applicable to different database platforms and hardware environments, further expanding its application scope.

[0049] In order to overcome the deficiencies of the prior art in query memory prediction and load scheduling, the present invention provides a query memory prediction and load scheduling method based on Graph Neural Network (GNN): MemQ. By combining the structured modeling ability of the graph neural network and the dynamic scheduling strategy, it solves the problems of inaccurate memory prediction and inefficient resource scheduling in complex query environments. It can achieve efficient memory management and scheduling optimization in complex database query scenarios. The present invention constructs a graph structure representation of the query plan, uses GNN to accurately predict the query memory requirements, and combines the dynamic scheduling strategy to achieve efficient resource allocation in a multi-query environment, significantly improving the query performance and reducing the waste of memory resources at the same time.

[0050] The database query prediction and load scheduling method provided by the present invention significantly improves the query efficiency and stability of the database system in various load scenarios by using the graph neural network for memory prediction and combining the memory-aware scheduling strategy. The combination of the above technologies not only solves the resource conflict problem of high-concurrency queries, but also provides new ideas for future database optimization, effectively improving the memory prediction accuracy and load scheduling efficiency of query tasks.

[0051] The database query prediction and load scheduling method described in the present invention can be applied to the visual analysis and processing process. In the intelligent monitoring and automation system corresponding to visual analysis, a large number of video streams and analysis results need to be processed, and the query tasks can be query tasks mainly based on timestamps and event types. Correspondingly, in the visual analysis and processing process, the types of multiple query tasks corresponding to the database system can be retrieving monitoring events by time range, for example: SELECT * FROM events WHERE timestamp BETWEEN '2025-01-01 00:00:00' AND '2025-01-01 23:59:59'; putting the monitoring events retrieved by time range into the event type statistical results, for example: SELECT event_type, COUNT(*) FROM events GROUP BY event_type.

[0052] In addition, the database query prediction and load scheduling method described in the present invention can also be applied to real-time recommendation systems. Real-time recommendation systems need to generate recommendation content in real time based on user behavior data, such as real-time recommendation systems in fields such as e-commerce and content distribution platforms. Correspondingly, the types of query tasks can be personalized recommendations based on user historical behavior, for example: SELECT product_id FROM user_behavior WHERE user_id = '5678' AND action = 'click'; the types of query tasks can also be counting real-time popular products, for example: SELECT product_id, COUNT(*) AS view_count FROM user_behavior WHERE action = 'view' GROUP BY product_id ORDER BY view_count DESC LIMIT 10.

[0053] The database query prediction and load scheduling method described in the present invention can also be applied to the query and analysis process of medical and health information. Electronic medical records and device monitoring data in the medical field need to be quickly queried and analyzed, such as patient data retrieval.

[0054] The types of multiple query tasks corresponding to the database system include: retrieving case records of a certain disease, for example: SELECT * FROM medical_records WHERE diagnosis = 'diabetes' AND age>40; The types of multiple query tasks corresponding to the database system also include querying the usage of specific drugs, for example: SELECT COUNT(*) FROM prescriptions WHERE drug_name = 'Aspirin'; Find patients diagnosed with diabetes within a specific time period, list the medications prescribed to them, and calculate the total medication cost for each patient: SELECT p.patient_id; p.name AS patient_name; d.drug_name,; COUNT(r.prescription_id) AS prescription_count; SUM(d.cost) AS total_cost; FROM patients p; JOIN diagnoses diag ON p.patient_id = diag.patient_id; JOIN prescriptions r ON p.patient_id = r.patient_id; JOIN drugs d ON r.drug_id = d.drug_id; WHERE diag.diagnosis = 'diabetes'; AND diag.diagnosis_date BETWEEN '2024-01-01' AND '2024-12-31'; GROUP BY p.patient_id, p.name, d.drug_name; ORDER BY total_cost DESC, prescription_count DESC。

[0055] It should be noted that the database query prediction and load scheduling method of the present invention can significantly optimize the usage efficiency of database resources in these scenarios. By accurately predicting the memory requirements of queries and dynamically adjusting the execution order, it not only avoids resource contention and performance bottlenecks, but also improves the stability and response speed of the system in high-concurrency and complex load environments, thus meeting the application requirements with extremely high real-time and complexity requirements such as visual analysis, real-time recommendation systems, and medical and health fields.

[0056] The database query prediction and load scheduling device provided by the present invention will be described below. The database query prediction and load scheduling device described below can be mutually corresponding and referred to with the database query prediction and load scheduling method described above. Refer to Figure 6 As shown, it is a schematic structural diagram of the database query prediction and load scheduling device provided by the present invention. The database query prediction and load scheduling device of the present invention specifically includes the following parts: A query task acquisition unit 601, configured to acquire a plurality of query tasks corresponding to a database system.

[0057] A memory prediction processing unit 602, configured to input the plurality of query tasks into a preset memory prediction model respectively, and obtain memory prediction results corresponding to the plurality of query tasks output by the memory prediction model; wherein, the memory prediction model; wherein, the memory prediction model is a heterogeneous graph neural network model trained based on a query plan and memory prediction result labels respectively corresponding to the query plan; the query plan is a plan corresponding to the query task; the memory prediction result is a memory requirement prediction value.

[0058] A load scheduling processing unit 603, configured to perform load scheduling on the plurality of query tasks based on the memory prediction results and a dynamic scheduling policy, obtain load scheduling information, and perform memory allocation processing on the plurality of query tasks based on the load scheduling information; wherein, the dynamic scheduling policy is a load scheduling policy based on memory awareness of the database system; the load scheduling information includes the query execution order of the plurality of query tasks.

[0059] According to the database query prediction and load scheduling device of the present invention, before acquiring a plurality of query tasks corresponding to a database system, it further includes: a model training unit; The model training unit is specifically configured to: Acquire a query plan corresponding to a database system; Perform a graph structure representation on the query plan to obtain a sample graph data structure corresponding to the query plan; based on the sample graph data structure, obtain an initial memory prediction model; Obtain a training set and a validation set based on the query plan and the memory prediction result tags corresponding to the query plan; wherein, the memory prediction result tags are the predicted memory requirement values corresponding to the query plan respectively. Based on the training set and the validation set, perform model training and validation on the initial memory prediction model to obtain a memory prediction model.

[0060] According to the database query prediction and load scheduling device of the present invention, the graph structure representation of the query plan to obtain the graph data structure corresponding to the query plan includes: Define the operators in the query plan as nodes in the graph data structure; wherein, the attributes of the nodes include the type of the operator, the estimated intermediate result size, and the data distribution. Define the data flow relationship between the operators in the query plan as the edges constituting the graph data structure; wherein, the attributes of the edges of the graph data structure include the connection condition and the number of rows of data transmission. Based on the nodes in the graph data structure and the edges of the graph data structure, obtain the corresponding graph data structure.

[0061] According to the database query prediction and load scheduling device of the present invention, the memory prediction processing unit is specifically used for: Perform feature representation on the multiple query tasks to obtain the graph data structure of the multiple query tasks. Extract features from the nodes in the graph data structure through the memory prediction model, and perform multi-layer propagation and aggregation on the node features to obtain the memory prediction results corresponding to the multiple query tasks respectively.

[0062] According to the database query prediction and load scheduling device of the present invention, the load scheduling processing unit is specifically used for: Model the multiple query tasks to obtain corresponding multiple query task models. Based on the memory prediction results and the dynamic scheduling policy, perform dynamic priority sorting on the multiple query task models to obtain the query execution order corresponding to the multiple query task models. Obtain the memory usage information of the database system; based on the memory usage information of the database system and the query execution order corresponding to the multiple query task models, perform memory allocation processing on the multiple query tasks.

[0063] According to the database query prediction and load scheduling device of the present invention, after performing memory allocation processing on the multiple query tasks, it further includes: The dynamic scheduling policy update processing unit is used for: Obtain the latest memory usage information of the database system, update the dynamic scheduling policy based on the new memory usage information, and obtain a new dynamic scheduling policy; Based on the memory prediction results of the next memory prediction cycle and the new dynamic scheduling policy, perform load scheduling on multiple query tasks input within the next memory prediction cycle to obtain new load scheduling information; Based on the new load scheduling information, perform memory allocation processing on multiple query tasks input within the next memory prediction cycle.

[0064] The database query prediction and load scheduling device provided by the present invention obtains multiple query tasks corresponding to the database system, inputs the multiple query tasks into a preset memory prediction model respectively, obtains the memory prediction results corresponding to the multiple query tasks output by the memory prediction model respectively, and performs load scheduling on the multiple query tasks based on the memory prediction results and the dynamic scheduling policy to obtain load scheduling information, and performs memory allocation processing on the multiple query tasks based on the load scheduling information. The dynamic scheduling policy is a load scheduling policy based on memory awareness of the database system; the load scheduling information includes the query execution order of multiple query tasks, which can effectively improve the memory prediction accuracy and load scheduling efficiency of query tasks.

[0065] Figure 7 An example of a schematic physical structure diagram of an electronic device is shown in Figure 7As shown in the figure, the electronic device may include: a processor 701, a communications interface 704, a memory 702, and a communication bus 703. Among them, the processor 701, the communications interface 704, and the memory 702 communicate with each other through the communication bus 703. The processor 701 may call logic instructions in the memory 702 to execute a database query prediction and load scheduling method, which includes: obtaining a plurality of query tasks corresponding to the database system; respectively inputting the plurality of query tasks into a preset memory prediction model to obtain memory prediction results corresponding to the plurality of query tasks output by the memory prediction model; where the memory prediction model; where the memory prediction model is a heterogeneous graph neural network model trained based on a query plan and a memory prediction result label corresponding to the query plan respectively; the query plan is a plan corresponding to the query task; the memory prediction result is a memory requirement prediction value; performing load scheduling on the plurality of query tasks based on the memory prediction result and a dynamic scheduling strategy to obtain load scheduling information, and performing memory allocation processing on the plurality of query tasks based on the load scheduling information; where the dynamic scheduling strategy is a load scheduling strategy based on memory awareness of the database system; the load scheduling information includes the query execution order of the plurality of query tasks.

[0066] In addition, when the logic instructions in the above-mentioned memory 702 are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0067] On the other hand, the present application also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the database query prediction and load scheduling method provided by the above-mentioned various methods. The method includes: obtaining a plurality of query tasks corresponding to a database system; respectively inputting the plurality of query tasks into a preset memory prediction model to obtain memory prediction results respectively corresponding to the plurality of query tasks output by the memory prediction model; wherein, the memory prediction model; wherein, the memory prediction model is a heterogeneous graph neural network model trained based on a query plan and memory prediction result labels respectively corresponding to the query plan; the query plan is a plan corresponding to the query task; the memory prediction result is a memory requirement prediction value; performing load scheduling on the plurality of query tasks based on the memory prediction result and a dynamic scheduling strategy to obtain load scheduling information, and performing memory allocation processing on the plurality of query tasks based on the load scheduling information; wherein, the dynamic scheduling strategy is a load scheduling strategy based on memory awareness of the database system; the load scheduling information includes the query execution order of the plurality of query tasks.

[0068] On another aspect, the present application also provides a computer-readable storage medium, which includes a stored program. When the program runs, it executes the database query prediction and load scheduling method provided by the above-mentioned various methods. The method includes: obtaining a plurality of query tasks corresponding to a database system; respectively inputting the plurality of query tasks into a preset memory prediction model to obtain memory prediction results respectively corresponding to the plurality of query tasks output by the memory prediction model; wherein, the memory prediction model; wherein, the memory prediction model is a heterogeneous graph neural network model trained based on a query plan and memory prediction result labels respectively corresponding to the query plan; the query plan is a plan corresponding to the query task; the memory prediction result is a memory requirement prediction value; performing load scheduling on the plurality of query tasks based on the memory prediction result and a dynamic scheduling strategy to obtain load scheduling information, and performing memory allocation processing on the plurality of query tasks based on the load scheduling information; wherein, the dynamic scheduling strategy is a load scheduling strategy based on memory awareness of the database system; the load scheduling information includes the query execution order of the plurality of query tasks.

[0069] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative effort.

[0070] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A database query prediction and load scheduling method, characterized in that: include: Obtain multiple query tasks corresponding to the database system; Input the multiple query tasks into a preset memory prediction model respectively, and obtain the memory prediction results output by the memory prediction model and corresponding to the multiple query tasks respectively; wherein the memory prediction model; wherein the memory prediction model is a heterogeneous graph neural network model trained based on the query plan and the memory prediction result labels corresponding to the query plan respectively; the query plan is a plan corresponding to the query task; the memory prediction result is a memory demand prediction value; Based on the memory prediction result and the dynamic scheduling strategy, the multiple query tasks are load scheduled to obtain load scheduling information, and memory allocation processing is performed on the multiple query tasks based on the load scheduling information; wherein the dynamic scheduling strategy is a load scheduling strategy based on memory awareness of the database system; the load scheduling information includes a query execution order of multiple query tasks.

2. The database query prediction and load scheduling method according to claim 1, characterized in that: Before obtaining multiple query tasks corresponding to the database system, it also includes: Obtain the query plan corresponding to the database system; Representing the query plan in a graph structure to obtain a sample graph data structure corresponding to the query plan; and obtaining an initial memory prediction model based on the sample graph data structure; Based on the query plan and the memory prediction result label corresponding to the query plan, a training set and a validation set are obtained; wherein the memory prediction result label is the memory demand prediction value corresponding to the query plan respectively; Based on the training set and the validation set, model training and validation are performed on the initial memory prediction model to obtain a memory prediction model.

3. The database query prediction and load scheduling method according to claim 2, characterized in that: The step of representing the query plan in a graph structure to obtain a graph data structure corresponding to the query plan includes: defining operators in the query plan as nodes in a graph data structure; wherein the attributes of the nodes include the type of the operator, the estimated size of the intermediate result, and the data distribution; Defining the data flow relationship of the operator quality check in the query plan as an edge constituting a graph data structure; wherein the attributes of the edge of the graph data structure include a connection condition and the number of rows of data transmission; Based on the nodes in the graph data structure and the edges of the graph data structure, a corresponding graph data structure is obtained.

4. The database query prediction and load scheduling method according to claim 1, characterized in that: The step of inputting the plurality of query tasks into a preset memory prediction model respectively, and obtaining memory prediction results output by the memory prediction model and corresponding to the plurality of query tasks respectively, comprises: Performing feature representation on the multiple query tasks to obtain graph data structures of the multiple query tasks; The memory prediction model is used to extract features of the nodes in the graph data structure, and the node features are propagated and aggregated in multiple layers to obtain memory prediction results corresponding to the multiple query tasks.

5. The database query prediction and load scheduling method according to claim 1, characterized in that: The performing load scheduling on the multiple query tasks based on the memory prediction result and the dynamic scheduling strategy, obtaining load scheduling information, and performing memory allocation processing on the multiple query tasks based on the load scheduling information specifically includes: Modeling the multiple query tasks to obtain corresponding multiple query task models; Based on the memory prediction result and the dynamic scheduling strategy, the multiple query task models are dynamically prioritized to obtain the query execution order corresponding to the multiple query task models; Acquire memory usage information of the database system; and perform memory allocation processing on the multiple query tasks based on the memory usage information of the database system and the query execution order corresponding to the multiple query task models.

6. The database query prediction and load scheduling method according to claim 5, characterized in that: After performing memory allocation processing on the multiple query tasks, the method further includes: Acquire the latest memory usage information of the database system, and update the dynamic scheduling strategy based on the new memory usage information to obtain a new dynamic scheduling strategy; Based on the memory prediction result of the next memory prediction cycle and the new dynamic scheduling strategy, load scheduling is performed on multiple query tasks input in the next memory prediction cycle to obtain new load scheduling information; Memory allocation processing is performed on multiple query tasks input within the next memory prediction cycle based on the new load scheduling information.

7. A database query prediction and load scheduling device, characterized in that: include: A query task obtaining unit, used to obtain multiple query tasks corresponding to the database system; A memory prediction processing unit, used to input the multiple query tasks into a preset memory prediction model respectively, and obtain memory prediction results output by the memory prediction model and corresponding to the multiple query tasks respectively; wherein the memory prediction model; wherein the memory prediction model is a heterogeneous graph neural network model trained based on a query plan and memory prediction result labels corresponding to the query plans respectively; the query plan is a plan corresponding to the query task; the memory prediction result is a memory demand prediction value; A load scheduling processing unit is used to perform load scheduling on the multiple query tasks based on the memory prediction results and the dynamic scheduling strategy, obtain load scheduling information, and perform memory allocation processing on the multiple query tasks based on the load scheduling information; wherein the dynamic scheduling strategy is a load scheduling strategy based on memory awareness of the database system; the load scheduling information includes the query execution order of the multiple query tasks.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the database query prediction and load scheduling method as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the database query prediction and load scheduling method as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the database query prediction and load scheduling method as described in any one of claims 1 to 6 is implemented.

Images