Query task scheduling method and device based on data processing engine, electronic equipment, readable storage medium and computer program product

Abstract

The invention provides a query task scheduling method and device based on a data processing engine, electronic equipment, a readable storage medium and a computer program product, and relates to the field of data processing, in particular to the field of data query task scheduling. According to the implementation scheme, based on an SQL query request, a data processing engine is utilized, a physical execution plan tree of the SQL query request is determined, the physical execution plan tree comprises at least one leaf node, each leaf node in the at least one leaf node represents description information of a reading operation of a data table corresponding to the SQL query request, and each leaf node in the at least one leaf node represents the description information of the reading operation of the data table corresponding to the SQL query request. The description information at least comprises a record of the number of partitions of the data table; based on the partition number of the data table corresponding to the at least one leaf node, determining the query task number of the SQL query request; determining a target resource pool of the SQL query request from a plurality of candidate resource pools based on the query task number; and executing scheduling of the query task of the SQL query request based on the target resource pool.

Description

Technical Field

[0001] This disclosure relates to the field of data processing, and more particularly to the field of data query task scheduling, specifically to a query task scheduling method, apparatus, electronic device, computer-readable storage medium, and computer program product based on a data processing engine. Background Technology

[0002] With the continuous development of big data technology, the demand for data analysis is increasing. In the field of big data processing technology, Apache Spark, as a mainstream engine, while improving task startup efficiency with its resident computing mode, faces a fundamental deficiency in intelligent resource scheduling. The existing architecture lacks preprocessing and resource demand prediction capabilities in the SQL query submission stage, resulting in resource allocation strategies that are seriously out of sync with the actual load. To address these issues, there is an urgent need for a dynamic scheduling mechanism that can accurately identify resource demands and achieve "query-as-optimization" to support the efficient and stable operation of query tasks under multi-tenant mixed loads. Summary of the Invention

[0003] This disclosure provides a query task scheduling method, apparatus, electronic device, computer-readable storage medium, and computer program product based on a data processing engine.

[0004] According to one aspect of this disclosure, a query task scheduling method based on a data processing engine is provided, comprising: determining a physical execution plan tree for an SQL query request using the data processing engine, wherein the physical execution plan tree includes at least one leaf node, and wherein each leaf node represents descriptive information of a read operation on a data table corresponding to the SQL query request, and wherein the descriptive information includes at least a record of the number of partitions of the data table; determining the number of query tasks for the SQL query request based on the number of partitions of the data table corresponding to the at least one leaf node; determining a target resource pool for the SQL query request from multiple candidate resource pools based on the number of query tasks; and scheduling the query tasks of the SQL query request based on the target resource pool.

[0005] According to a second aspect of this disclosure, a query task scheduling apparatus based on a data processing engine is provided, comprising: a physical execution plan tree determination module, configured to determine a physical execution plan tree for an SQL query request based on the data processing engine, wherein the physical execution plan tree includes at least one leaf node, and wherein each leaf node represents descriptive information of a read operation on a data table corresponding to the SQL query request, and wherein the descriptive information includes at least a record of the number of partitions of the data table; a query task number determination module, configured to determine the number of query tasks for the SQL query request based on the number of partitions of the data table corresponding to the at least one leaf node; a target resource pool determination module, configured to determine a target resource pool for the SQL query request from multiple candidate resource pools based on the number of query tasks; and a query task execution module, configured to schedule the query tasks of the SQL query request based on the target resource pool.

[0006] According to a third aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described query task scheduling method.

[0007] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the query task scheduling method as described above.

[0008] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the above-described query task scheduling method.

[0009] According to one or more embodiments of this disclosure, a physical execution plan tree for an SQL query request is generated through a data processing engine. Before the user's SQL query task is executed, the number of query tasks for the SQL query request is determined based on the number of partitions in the leaf section of the physical execution plan tree. Based on the relationship between the determined number of query tasks and at least one pre-determined threshold number of tasks, the target resource pool for the SQL query request is determined from multiple candidate resource pools. The scheduling of query tasks is performed based on the target resource pool, thereby improving query efficiency and optimizing resource utilization in an overall balanced manner.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0012] Figure 1 A schematic diagram of an exemplary system in which the various methods described herein may be implemented according to embodiments of the present disclosure is shown; Figure 2 A flowchart of a query task scheduling method based on a data processing engine according to an embodiment of the present disclosure is shown; Figure 3 A flowchart of a query task scheduling method based on a data processing engine according to an embodiment of the present disclosure is shown; Figure 4 A block diagram of a query task scheduling apparatus based on a data processing engine according to an embodiment of the present disclosure is shown; Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0013] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0014] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0015] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0016] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0017] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0018] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the execution of content recommendation methods.

[0019] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105 and / or 106 under a Software as a Service (SaaS) model.

[0020] exist Figure 1 In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0021] Users can use client devices 101, 102, 103, 104, 105, and / or 106 to recommend content. The client devices can provide an interface that allows users to interact with the client devices. The client devices can also output information to users through this interface. Although... Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0022] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0023] Network 110 can be any type of network well known to those skilled in the art, and can support data communication using any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.). By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0024] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0025] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0026] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105 and / or 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105 and / or 106.

[0027] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0028] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0029] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0030] Figure 1The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0031] Figure 2 This is a flowchart illustrating a query task scheduling method 200 according to an embodiment of the present disclosure, such as... Figure 2 As shown, the query task scheduling method may include: step S202, determining the physical execution plan tree of the SQL query request using the data processing engine based on the SQL query request, wherein the physical execution plan tree includes at least one leaf node, and wherein each leaf node in the at least one leaf node represents descriptive information of the read operation of the data table corresponding to the SQL query request, and wherein the descriptive information includes at least a record of the number of partitions of the data table; step S204, determining the number of query tasks of the SQL query request based on the number of partitions of the data table corresponding to the at least one leaf node; step S206, determining the target resource pool of the SQL query request from multiple candidate resource pools based on the number of query tasks; and step S208, scheduling the query tasks of the SQL query request based on the target resource pool.

[0032] In this embodiment, a physical execution plan tree for SQL query requests is generated through a data processing engine. Before the user's SQL query task is executed, the number of query tasks for the SQL query request is determined based on the number of partitions in the leaf section of the physical execution plan tree. Based on the relationship between the determined number of query tasks and at least one pre-determined threshold number of tasks, the target resource pool for the SQL query request is determined from multiple candidate resource pools. The scheduling of query tasks is then performed based on the target resource pool, thereby improving query efficiency and optimizing resource utilization in an overall balanced manner.

[0033] In some embodiments, the data processing engine (such as Apache Spark, Flink, Hive, Impala, etc.) is the core software component or system responsible for executing data processing tasks. Its main responsibility is to receive a high-level instruction (such as an SQL query), parse and optimize it, generate an executable plan, and finally schedule resources to complete the reading, transformation, calculation, and output of data. The data processing engine includes at least the following core components: Query parser: Used to convert the received SQL query string into an abstract syntax tree (AST); and to check the correctness of the SQL syntax; The analyzer performs semantic analysis on the AST, including verifying the existence of table names and column names, and checking data type matching; it uses metadata to parse database objects; and it outputs a parsed logical plan. Logical plan optimizer: Applies a series of optimization rules to the logical plan, such as predicate pushdown, constant folding, column pruning, and join reordering; and outputs an optimized logical plan. Physical plan generator: converts the optimized logical plan into a physical execution plan tree; the physical execution plan describes how to specifically execute operations, such as what algorithms to use for joins, how to partition the data, etc.; and Execution Engine: Used to convert physical plans into a series of tasks that can be executed on the cluster; and to manage task scheduling, execution, fault tolerance, etc.

[0034] In some embodiments, determining the physical execution plan tree of the SQL query request using the data processing engine based on the SQL query request includes: step S302, converting the SQL query request into an abstract syntax tree and checking the correctness of the syntax and semantics in the abstract syntax tree; step S304, converting the AST into an initial logical execution plan, including operations such as scanning, filtering, and joining the data table corresponding to the SQL query request; step S306, using optimization rules such as predicate pushdown, column pruning, and cost optimization to convert the initial logical plan into an optimized logical plan; and step S308, converting the optimized logical plan into a physical execution plan tree, wherein the physical execution plan tree defines how to execute the query operation, including specific steps for data reading, transformation, and writing.

[0035] The physical execution plan tree can be understood as a blueprint for executing query tasks corresponding to SQL query requests. It consists of multiple nodes, each representing a physical operation (such as scanning, aggregation, sorting, etc.). It includes a root node, intermediate nodes, and leaf nodes. The root node represents the output operation of the final query result (such as data collection or writing), intermediate nodes represent various transformation operations (such as data table filtering, joining, grouping, and aggregation), and when an intermediate node has child nodes, it also processes data from those child nodes. Leaf nodes represent read operations on the data table. In distributed systems, the physical execution plan tree is designed for parallel execution, and leaf nodes typically divide the data table into different partitions to achieve load balancing and high performance.

[0036] In some embodiments, each leaf node contains descriptive information about data table read operations, which guides the data engine on how to access the data table. This descriptive information includes at least: Data table identifier: such as table name, database name, or file path; Read methods: such as full table scan, index scan, or partition scan; and Partition count record: Indicates the number of partitions a data table is divided into. In distributed storage, tables are usually divided into multiple partitions (e.g., partitions based on date, region, or other keys). Each partition is an independent data block, and each partition corresponds to an independent query task.

[0037] In some embodiments, the leaf node can be determined by recursively traversing the physical execution plan tree using depth-first search (DFS). The specific steps are as follows: starting from the root node, check each node layer by layer downwards; if the current node is in the optimization layer, skip the node and directly recursively traverse its internal input plan; obtain a list of all child nodes of the current node; if child nodes exist, repeat the recursive traversal for each child node; if there are no child nodes, determine it as a leaf node, and collect its column attributes and execution methods (such as the output fields and partitioning logic of file scan operations); when a node has no child nodes, confirm it as a leaf node. It should be noted that the method of determining leaf nodes by recursively traversing the physical execution plan tree using depth-first search (DFS) is only an example, and other methods that can effectively determine leaf nodes can also be used, without limitation.

[0038] Based on the above embodiments, the number of partitions in step S204 directly determines the number of parallel query tasks for the read operation. Each partition can be processed by a single query task (such as a Spark task), thereby utilizing the multi-node parallel computing of the cluster. For example, if the table has 100 partitions, then up to 100 tasks can be started to read data simultaneously.

[0039] In some embodiments, for scenarios such as databases, data warehouses, data lakes, and stream processing platforms that require processing structured or semi-structured data queries, the technical solutions of the above embodiments can efficiently utilize computing resources to quickly return query results.

[0040] In some embodiments, determining the number of query tasks for the SQL query request based on the number of partitions of the data tables corresponding to all leaf nodes in the at least one leaf node includes: taking the sum of the number of partitions of the data tables corresponding to all leaf nodes as the number of query tasks for the SQL query request.

[0041] In this embodiment, the number of tasks is determined by the number of data partitions, ensuring that each data partition can be processed independently in parallel, improving the fine-grainedness of query task execution, and providing the basis for optimal parallelism in query task execution.

[0042] In some embodiments, determining the target resource pool for the SQL query request from multiple candidate resource pools based on the number of query tasks includes: determining the target resource pool from multiple candidate resource pools based on the relationship between the number of query tasks and a first task number threshold and a second task number threshold, wherein the first task number threshold is less than the second task number threshold.

[0043] In this embodiment, the target resource pool is determined from multiple candidate resource pools based on the relationship between the number of query tasks and the first and second task thresholds, thereby further improving the adaptability of the target resource pool to the SQL query request.

[0044] In some embodiments, the first task number threshold and the second task number threshold are two preset values ​​that serve as a benchmark for determining the target resource pool; the role of these two thresholds is to transform the abstract "query scale" into a specific, operable scheduling decision.

[0045] In some embodiments, the number of query tasks, actual execution time, CPU and memory consumption of historical queries can be statistically analyzed periodically or under triggered conditions, and the first task number threshold and the second task number threshold can be dynamically updated based on the statistical results to achieve optimal scheduling performance. For example, the number of query tasks, actual execution time, CPU and memory consumption of historical queries, and the updates to the first and second task number thresholds can be performed at the time of lowest load (e.g., 3:00 AM every day). The updates can also be triggered when there are significant changes in the computing resources of the data processing engine, such as when a new computing node is added to the cluster or a computing node is removed from the cluster.

[0046] In some embodiments, determining the target resource pool from multiple candidate resource pools based on the relationship between the number of query tasks and the first task number threshold and the second task number threshold includes: in response to determining that the number of query tasks is less than the first task number threshold, selecting the first resource pool from the multiple candidate resource pools as the target resource pool, wherein the first resource pool is a thread pool.

[0047] In this embodiment, when the number of query tasks is less than the first task number threshold, the first resource pool (thread pool) is selected for scheduling first, providing a low-overhead thread pool for small-scale queries, reducing scheduling overhead, and shortening the response time for small-scale queries.

[0048] In some embodiments, the thread pool maintains a set of pre-created threads for executing submitted tasks. Threads in the thread pool can be reused, thus avoiding the overhead of frequently creating and destroying threads. The thread pool is used to execute queries when the number of tasks (i.e., the number of partitions) is less than a first task threshold (e.g., 200). This is because these queries involve fewer partitions, smaller data volumes, and lower computational complexity, making them suitable for fast processing using a thread pool. For example, if a query needs to scan 30 partitions, it will generate 30 tasks. Since the number of tasks is less than the threshold of 200, these tasks will be submitted to the thread pool for execution. The thread pool may use 30 threads to process these tasks in parallel (if the thread pool has enough threads), thereby quickly completing the query.

[0049] In some embodiments, the scheduling of the query task for executing the SQL query request based on the target resource pool includes: scheduling the query task for executing the SQL query request based on the resources of the first resource pool using a first-in-first-out scheduling algorithm.

[0050] In this embodiment, to accommodate small-scale queries, a first-in-first-out (FIFO) scheduling algorithm is used in the first resource pool to provide a low-latency execution order and reduce scheduling complexity.

[0051] In some embodiments, the First-In-First-Out (FIFO) scheduling algorithm refers to executing query tasks in the order they arrive. In a thread pool, tasks are submitted to a queue, and worker threads in the thread pool retrieve tasks from the queue sequentially and execute them. It's important to note that in actual thread pool implementations, the query task queue is typically a FIFO queue. However, while the worker threads in the thread pool retrieve tasks from the queue concurrently, the queue itself guarantees a FIFO order; however, in distributed query task scheduling, each query request may be broken down into multiple tasks.

[0052] In some embodiments, determining the target resource pool from multiple candidate resource pools based on the relationship between the number of query tasks and the first task number threshold and the second task number threshold includes: in response to determining that the number of query tasks is greater than or equal to the first task number threshold and less than the second task number threshold, selecting the second resource pool from the multiple candidate resource pools as the target resource pool, wherein the second resource pool is the default resource pool of the data processing engine.

[0053] In this embodiment, suitable for medium-sized queries, when the number of query tasks is greater than or equal to the first threshold number of tasks and less than the second threshold number of tasks, the second resource pool (the default resource pool of the data processing engine) is selected to ensure the stability of query task execution.

[0054] In some embodiments, the default resource pool is a pre-configured set of core computing resources in the data processing engine for handling medium-sized query tasks; it undertakes most of the regular data processing workloads, achieving an optimal balance between performance, resources, and cost.

[0055] In some embodiments, the scheduling of the query task for executing the SQL query request based on the target resource pool includes: scheduling the query task for executing the SQL query request based on the resources of the second resource pool using the default scheduling algorithm of the data processing engine.

[0056] In this embodiment, suitable for medium-sized queries, scheduling is based on the default scheduling algorithm of the data processing engine, making full use of the engine's built-in algorithm features to improve the throughput and resource utilization of medium-sized queries.

[0057] For example, an e-commerce company needs to generate daily sales reports. The data processing engine determines the number of partitions that the leaf nodes of the physical execution plan tree need to scan. The order table is partitioned by date, with the query condition being the most recent day, so 10 partitions are scanned; the order details table is partitioned by order, totaling 500 partitions; the product table is partitioned by category, totaling 100 partitions; the region table is scanned entirely, requiring 10 partitions. The total number of tasks is 10 + 500 + 100 + 10 = 620, ultimately calculated as 620 tasks. Since the number of tasks (620) is greater than or equal to the first task threshold of 200 and less than the second task threshold of 2000, the second resource pool (the default resource pool) is selected for execution. It should be noted that the above examples of the first and second task thresholds are merely illustrative and not restrictive; the specific values ​​should be pre-determined based on the actual application scenario.

[0058] For the default scheduling algorithm of a data processing engine, taking Apache Spark as an example, Apache Spark's default scheduling algorithm is based on a scheduler. In Spark, there are various schedulers that can use corresponding scheduling strategies to schedule query tasks, including: First-In-First-Out (FIFO) scheduler: Schedules tasks according to the order in which they are submitted.

[0059] Fair scheduling allows multiple jobs to share cluster resources and ensures that all query tasks receive fair resource allocation as much as possible.

[0060] Capacity scheduling: Provides capacity limiting functionality for resource pools; each resource pool can be configured with a maximum resource usage, suitable for scenarios where it is necessary to guarantee a specific proportion of resources for certain queries or users.

[0061] Isolation Workload Manager: Provides a more granular mechanism for resource isolation and sharing; by defining different resource groups, it allows for more flexible management of resource usage by different task queries or user groups.

[0062] In this embodiment, the scheduler corresponding to the default scheduling algorithm of the data processing engine used after selecting the second target resource pool can be preset according to the application scenario, or the default selection mechanism of the data processing engine can be used, and there is no limitation on this.

[0063] In some embodiments, determining the target resource pool from multiple candidate resource pools based on the relationship between the number of query tasks and the first task number threshold and the second task number threshold includes: in response to determining that the number of query tasks is greater than or equal to the second task number threshold, selecting a third resource pool from the multiple candidate resource pools as the target resource pool, wherein the third resource pool is an elastic resource pool.

[0064] In this embodiment, to accommodate large-scale queries, when the number of query tasks is greater than or equal to the second threshold, the system switches to the third resource pool (elastic resource pool) to provide greater / more dynamic concurrency capabilities through elastic resources, thereby reducing the waiting time of tasks to be executed.

[0065] In some embodiments, the query task queue status of the database service is monitored in real time for queries involving massive amounts of data (such as full table scans and multi-table joins). For example, when a user initiates a full table scan involving a TB-level dataset or a complex join operation across multiple dimension tables, such high computational density queries often cause a rapid backlog in the task queue. For instance, in the daily batch processing jobs of a financial risk control system, a full table scan operation generates a large number of concurrent requests, causing the number of query tasks to instantly reach 52,000 (exceeding the preset second task number threshold of 50,000). At this time, the system makes a judgment based on the relationship between the number of query tasks and the threshold: in response to determining that the number of query tasks is greater than or equal to the second task number threshold, the third resource pool (elastic resource pool) is directly used as the target resource pool. The specific process is as follows: the current task number is collected every 10 seconds. If the number of tasks is detected to be continuously ≥50,000 (the second task number threshold), the system immediately switches to the elastic resource pool and dynamically expands the cluster—assuming an initial allocation of 15 computing nodes, 3 computing nodes are automatically added for every 100 query tasks exceeding the threshold, ensuring that the response time of a single full table scan query remains stable within the set time. The basis for this choice is that in scenarios with massive queries, the fixed resource pool (first resource pool) is prone to triggering queuing timeouts due to the capacity limit (100 tasks), and the pre-allocated pool (second resource pool) cannot cope with sudden loads. In contrast, the elastic resource pool effectively avoids task backlog by replenishing resources in seconds, thereby improving the throughput of query tasks in scenarios with multiple table joins.

[0066] In some embodiments, the scheduling of the query task for executing the SQL query request based on the target resource pool includes: scheduling the query task for executing the SQL query request based on the resources of the third resource pool using a time-slice round-robin scheduling algorithm.

[0067] In this embodiment, a time-slice round-robin scheduling algorithm is adopted for the third resource pool to ensure that query tasks receive a fair time slice and balance the waiting time of each query task.

[0068] In some embodiments, during e-commerce platform promotional events, the system monitors the database query task queue in real time, setting a first task threshold of 10,000 and a second task threshold of 50,000. When users initiate a large number of order queries (such as multi-table joins across inventory and transaction tables), resulting in 52,000 query tasks (greater than or equal to the second task threshold), the third resource pool (elastic resource pool) is automatically identified as the target resource pool. This elastic resource pool is dynamically managed by the cluster, initially configured with 20 compute nodes, and supports automatic scaling up and down based on load. Subsequently, SQL query tasks are scheduled based on the target resource pool, i.e., the scheduling module uses a time-slice round-robin algorithm to process the task queue. Specifically, all SQL query requests are stored in the ready queue in the order of arrival; each task is allocated a fixed time slice (100ms) and distributed to idle compute nodes for execution; if a task is not completed within the time slice (e.g., a full table scan operation takes 200ms), it is paused and moved to the end of the queue to wait for the next round of scheduling, releasing the node for new tasks. For example, a complex join query might be suspended after 100ms in the first round, continuing in the next round until completion, while lightweight queries are processed instantly. Simultaneously, the system assesses the load every 10 seconds; if the number of tasks consistently exceeds 50,000, the elastic pool automatically expands (adding 3 nodes for every 100 tasks exceeding the threshold) to ensure sufficient scheduling resources. The core rationale for choosing an elastic resource pool in this design is that its dynamic capacity can adapt to high concurrency demands, avoiding task backlogs caused by resource depletion in a fixed pool; the time-slice round-robin mechanism ensures fair scheduling of query tasks, preventing single long-running operations (such as TB-level data scanning) from blocking the queue; and compared to the priority scheduling of a static pool, the collaboration between time-slice round-robin and the elastic pool significantly optimizes service stability under sudden traffic surges.

[0069] In some embodiments, the method for scheduling the query task of the SQL query request based on the target resource pool includes: scheduling the query task of the SQL query request based on the resources of the third resource pool using a preemptive priority algorithm.

[0070] In this embodiment, corresponding to the third resource pool, preemptive priority scheduling is used, which enables high-priority tasks to preempt the execution of low-priority tasks when necessary, thereby improving the responsiveness to urgent tasks.

[0071] For example, in an embodiment of a bank's real-time risk control system, a first task threshold is set at 800 (the upper limit of the fixed resource pool capacity), and a second task threshold is set at 4500 (a high-load trigger point). The system continuously monitors the database query queue. When users intensively initiate transaction verification (high priority) and anti-fraud analysis (low priority) requests, the task count instantly reaches 4600 (less than the second task threshold), and the pre-allocated resource pool is activated. However, if a promotional traffic peak causes the task count to surge to 5100 (greater than or equal to the second task threshold of 4500), based on the relationship between the number of query tasks and the threshold, the third resource pool (elastic resource pool) is immediately identified as the target resource pool. The initial configuration includes 12 computing nodes and supports dynamic scaling.

[0072] Based on the target resource pool, the specific execution method for scheduling SQL query tasks is as follows: The scheduling module uses a preemptive priority algorithm to classify tasks. First, a static priority of 9 (highest level) is assigned to real-time transaction queries, and a priority of 3 (lowest level) is assigned to batch analysis queries. Second, when a high-priority task arrives, it immediately preempts the execution resources of a low-priority task—for example, if an anti-fraud analysis task (priority 3) is running, its execution is immediately interrupted upon encountering a new payment verification request (priority 9), releasing CPU resources for high-priority tasks to use, ensuring that payment verification is completed within 50ms. Simultaneously, the task status is saved to a memory buffer, and execution resumes after the high-priority task completes. The system checks the load every 15 seconds. If the number of tasks consistently exceeds 4500, the elastic pool is expanded by 1 node for every 500 tasks exceeding the threshold (e.g., expanding to 14 nodes for 5100 tasks) to ensure sufficient scheduling resources.

[0073] The core basis for choosing an elastic resource pool is that its dynamic capacity can meet the sudden resource demand of preemptive scheduling, avoiding the delay of critical tasks due to resource rigidity in a fixed pool; and the preemption mechanism ensures that high-value queries are processed first.

[0074] In some embodiments, the plurality of candidate resource pools are isolated from each other. In this embodiment, the isolation between the plurality of candidate resource pools ensures that the resource usage and corresponding query task scheduling status of any resource pool will not interfere with other resource pools, thereby improving the stability of query task scheduling.

[0075] For example, in an intelligent transportation management system, the first task threshold is set to 80 (corresponding to the upper limit of the fixed resource pool capacity), and the second task threshold is set to 400 (a high-load trigger point). The system monitors the query task queue of the urban traffic database in real time. Each candidate resource pool adopts a network isolation architecture: the first resource pool (thread pool), the second resource pool (the default resource pool for the data processing engine), and the third resource pool (elastic resource pool) are strictly isolated through firewall policies, prohibiting cross-pool data access and resource calls to ensure fault domain separation. When a sudden surge of real-time vehicle trajectory analysis requests occurs during holidays (such as multi-table join operations), and the number of query tasks reaches 410 (greater than or equal to the second task threshold), the system immediately determines the third resource pool as the target resource pool based on the relationship between the number of query tasks and the threshold. The specific process is as follows: after the scheduling module detects that the number of tasks is ≥400, it automatically switches to the elastic resource pool. The cluster initially allocates 12 computing nodes, and dynamically adds 1 node for every 50 tasks exceeding the threshold; all SQL query requests are processed independently by this pool without interacting with other pools. The basis for this choice is that, under the design of mutual isolation of resource pools, the expansion of the elastic pool only consumes dedicated resources, avoiding traffic signal control delays caused by resource contention in the first resource pool. At the same time, the isolation mechanism blocks the propagation of anomalies—for example, the failure of an elastic pool node does not affect the emergency scheduling service of the first resource pool.

[0076] It should be noted that the thresholds and other numerical values ​​in the above embodiments are only illustrative examples and not intended as exemplary descriptions.

[0077] Figure 4 This is a block diagram illustrating a query task scheduling apparatus 400 based on a data processing engine according to an embodiment of the present disclosure. In some embodiments, such as... Figure 4As shown, the query task scheduling device includes: a physical execution plan tree determination module 402, used to determine the physical execution plan tree of the SQL query request based on the SQL query request and using the data processing engine, wherein the physical execution plan tree includes at least one leaf node, and each leaf node represents descriptive information of the read operation of the data table corresponding to the SQL query request, and wherein the descriptive information includes at least a record of the number of partitions of the data table; a query task number determination module 404, used to determine the number of query tasks of the SQL query request based on the number of partitions of the data table corresponding to the at least one leaf node; a target resource pool determination module 406, used to determine the target resource pool of the SQL query request from multiple candidate resource pools based on the relationship between the number of query tasks and at least one task number threshold, wherein the at least one task number threshold is a configurable parameter preset according to the resource scale of the data processing engine; and a query task execution module 408, used to schedule the query tasks of the SQL query request based on the target resource pool.

[0078] It should be noted that, Figure 4 The various modules of the device 400 shown can be connected to the reference. Figure 2 The steps in method 200 described correspond to each other. Therefore, the operations, features, and advantages described above for method 200 also apply to apparatus 400 and its included modules and units. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0079] According to embodiments of this disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0080] refer to Figure 5 The present invention describes a structural block diagram of an electronic device 500 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0081] like Figure 5As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0082] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, output unit 507, storage unit 508, and communication unit 509. Input unit 506 can be any type of device capable of inputting information to electronic device 500. Input unit 506 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 507 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 508 may include, but is not limited to, disk and optical disk. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0083] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as method 200. For example, in some embodiments, method 200 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of method 200 described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform method 200 by any other suitable means (e.g., by means of firmware).

[0084] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0085] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0086] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0087] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0088] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0089] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0090] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0091] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A method for scheduling query tasks based on a data processing engine, comprising: determining, based on a SQL query request, a physical execution plan tree of the SQL query request by using the data processing engine, wherein the physical execution plan tree comprises at least one leaf node, and wherein each of the at least one leaf node represents description information of a read operation on a data table corresponding to the SQL query request, and wherein the description information comprises at least a record of a number of partitions of the data table; determining a number of query tasks of the SQL query request based on the number of partitions of the data table corresponding to the at least one leaf node; determining a target resource pool of the SQL query request from a plurality of candidate resource pools based on the number of query tasks; and performing scheduling of the query tasks of the SQL query request based on the target resource pool.

2. The query task scheduling method of claim 1, wherein, The determining of the number of query tasks of the SQL query request based on the number of partitions of the data table corresponding to the at least one leaf node comprises: taking a sum value of the number of partitions of the data table corresponding to the at least one leaf node as the number of query tasks of the SQL query request.

3. The query task scheduling method according to claim 1 or 2, wherein, The determining of the target resource pool of the SQL query request from the plurality of candidate resource pools based on the number of query tasks comprises: determining the target resource pool from the plurality of candidate resource pools based on a size relationship between the number of query tasks and a first number threshold and a second number threshold, wherein the first number threshold is smaller than the second number threshold.

4. The query task scheduling method of claim 3, wherein, The determining of the target resource pool from the plurality of candidate resource pools based on the size relationship between the number of query tasks and the first number threshold and the second number threshold comprises: in response to determining that the number of query tasks is smaller than the first number threshold, taking a first resource pool in the plurality of candidate resource pools as the target resource pool, wherein the first resource pool is a thread pool.

5. The method of claim 4, wherein, The performing of the scheduling of the query tasks of the SQL query request based on the target resource pool comprises: performing the scheduling of the query tasks of the SQL query request by using a first-in-first-out scheduling algorithm based on resources of the first resource pool.

6. The query task scheduling method according to any one of claims 3-5, wherein, The determining of the target resource pool from the plurality of candidate resource pools based on the size relationship between the number of query tasks and the first number threshold and the second number threshold comprises: in response to determining that the number of query tasks is greater than or equal to the first number threshold and smaller than the second number threshold, taking a second resource pool in the plurality of candidate resource pools as the target resource pool, wherein the second resource pool is a default resource pool of the data processing engine.

7. The method of claim 6, wherein, The performing of the scheduling of the query tasks of the SQL query request based on the target resource pool comprises: performing the scheduling of the query tasks of the SQL query request by using a default scheduling algorithm of the data processing engine based on resources of the second resource pool.

8. The query task scheduling method according to any one of claims 3-7, wherein, The determining of the target resource pool from the plurality of candidate resource pools based on the size relationship between the number of query tasks and the first number threshold and the second number threshold comprises: In response to determining that the query task number is greater than or equal to the second task number threshold, a third resource pool in the plurality of candidate resource pools is determined as the target resource pool, where the third resource pool is an elastic resource pool.

9. The method of claim 8, wherein, The scheduling of the query task of the SQL query request based on the target resource pool includes: The scheduling of the query task of the SQL query request is performed based on resources of the third resource pool using a time slice round robin scheduling algorithm.

10. The method of claim 8, wherein, The scheduling method of the query task of the SQL query request based on the target resource pool includes: The scheduling of the query task of the SQL query request is performed based on resources of the third resource pool using a preemptive priority algorithm.

11. The query task scheduling method according to any one of claims 1-10, wherein, The plurality of candidate resource pools are isolated from each other.

12. A data processing engine-based query task scheduling apparatus, comprising: a physical execution plan tree determination module configured to determine, based on a SQL query request, a physical execution plan tree of the SQL query request using the data processing engine, wherein the physical execution plan tree includes at least one leaf node, and wherein each of the at least one leaf node represents description information of a read operation on a data table corresponding to the SQL query request, and wherein the description information includes at least a record of a partition number of the data table; a query task number determination module configured to determine a query task number of the SQL query request based on the partition number of the data table corresponding to the at least one leaf node; a target resource pool determination module configured to determine, based on the query task number, a target resource pool of the SQL query request from a plurality of candidate resource pools; and a query task execution module configured to perform scheduling of a query task of the SQL query request based on the target resource pool.

13. An electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

14. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-11.

15. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-11. The computer program, when executed by a processor, implements the method of any one of claims 1-11.

Images